MOSA Runtime, Kernel, and Plugging

Developer
Apr 5, 2009 at 7:22 AM

Hey folks. This e-mail is about the direction we need to move in with the runtime and getting managed code to be both supported by, and supporting, our kernel. The beginning starts out as a point of reference for the less informed, but the end poses (and implies) some major discussions we need to have.

For those of you familiar with Cosmos’ or SharpOS’ experiences, you’ll know that the biggest problem with integrating the BCL is the fact that even Mono’s BCL has VM internal calls. Consider the following from the System.Environment class in Mono’s mscorlib:

For those of you not familiar, this is implemented by Mono in their runtime code, written in C:

/* note: we better manipulate the string in managed code (easier and safer) */
MonoString*
ves_icall_System_Environment_GetOSVersionString (void)
{
#ifdef PLATFORM_WIN32
        OSVERSIONINFO verinfo;
 
        MONO_ARCH_SAVE_REGS;
 
        verinfo.dwOSVersionInfoSize = sizeof (OSVERSIONINFO);
        if (GetVersionEx (&verinfo)) {
               char version [64];
               /* maximum string length is 35 bytes 
                  3 x 10 bytes per number, 1 byte for 0, 3 x 1 byte for dots, 1 for NULL */
               sprintf (version, "%ld.%ld.%ld.0", 
                       verinfo.dwMajorVersion,
                       verinfo.dwMinorVersion,
                       verinfo.dwBuildNumber);
               return mono_string_new (mono_domain_get (), version);
        }
#elif defined(HAVE_SYS_UTSNAME_H)
        struct utsname name;
 
        MONO_ARCH_SAVE_REGS;
 
        if (uname (&name) >= 0) {
               return mono_string_new (mono_domain_get (), name.release);
        }
#endif
        return mono_string_new (mono_domain_get (), "0.0.0.0");
}

Note, it’s not pretty. Its C. And I propose we can do better:

namespace MOSA.Runtime.Plugs.InternalSystem {

     

      [TypeOverridePlug(OriginalType=typeof(global::System.Environment))]

      public sealed abstract class Environment {

           

            [MethodOverridePlug]

            private MOSA.InternalSystem.String GetOSVersionString() {

            return MOSA.Runtime.RuntimeBase.CastClass<string, MOSA.InternalSystem.Environment> ("MOSA [1.0] Compliant OS");

            }

           

      }

}

Now obviously the string can come from anywhere. In my proposed system of Plugs (to coin a Cosmos term, though SharpOS does this too), we implement the icalls in one assembly, assembly ‘X’, by mirroring the types and using the structure demonstrated above in assembly ‘MOSA.Runtime.Plugs.X’. When the implementations inside of the plugs for one type reference (either by inheritance or by signature) internal members from another type from the same assembly ‘X’, we also include a skeleton of the referenced type in the Plug assembly. (There are more details to this convention, such as incorporation of fields and methods that don’t need to be overrided, that I can explain in a later post or IRC, as requested. See the output of my PlugGen tool, from trunk, if you are curious.) All of this would be wired up using features that I propose we plan to put into the compiler.

Attached, (or below, depending on your email client), is an attempted diagram of the beginnings of how I envision the “Plugging” process to fit into the MOSA Reference Kernel and Runtime.

cid:image001.png@01C9B57E.4A0B2180

The thin arrowed-lines indicate a standard assembly reference, and from one assembly to the referenced assembly being ‘pointed’ to. There are some possible references that I have not included, that we need to discuss, along with other tantalizing issues:

· As already mentioned, the compiler needs to support this. SharpOS did it with linker tricks, and the same could definitely be done with MOSA. Alternatively, we could extend the IR itself to be aware. (And then filter out with a specific compiler stage.) Regardless, note that a plug type (the type doing the ‘plugging’), is never included or accessible by runtime metadata. (Neither is its source assembly.) Despite the fancy decorations, at compile time, we would literally just be replacing an extern method with a method of the exact same signature except it has an implementation.

o This implies that if a Plug assembly has other references, they need to be included regardless

§ Which means we need to talk about the runtime exposure of such referenced assemblies

· Some types already in MOSA.Runtime would actually be moved to MOSA.Runtime.Plugs.mscorlib , such as System.Object. The problem is drawing a line between what the runtime needs to function and what the corlib Plug Assembly needs to function. (Do we need a third assembly to serve as a common reference for both?)

o The types that MOSA already has to manage metadata for runtime information come to mind

· We can use this methodology to override methods and types from Mono BCL or other assemblies that don’t even have icalls, in order to apply our own implementation of something like WinForms, while still untouching most of Mono’s tried and true codebase

· Don’t forget that different processes (including the kernel), or different MOSA Operating Systems themselves, may use different plug implementations without actual BCL changes. This allows for alot of flexability.

This has me very excited. I look forward to your responses, (questions, suggestions, and criticisms alike), on the MOSA Discussions list.

Bruce Markham

aka “illuminus”

Developer
Apr 5, 2009 at 7:25 AM
I forgot it would filter my attached image. Here's a link: http://picasaweb.google.com/lh/photo/pMX7u6CR-DvfTtLAoWeRdA?feat=directlink
Coordinator
Apr 5, 2009 at 6:12 PM

Thanks Illuminus. That was well thought out.

Just to explore another idea, would it be more advantageous to merge the mscorlib with plugs and produce a patched mscorlib? This way there is no plug assembly at runtime. And the patched mscorlib (because of the plugs) reference methods and types in the MOSA.Runtime (or others) as necessary.  Just a thought for discussion (I'm not advocating it).
-Phil
Developer
Apr 8, 2009 at 5:25 AM
I'd personally shy away now from doing a patched mscorlib. It will most
likely introduce hard-to-trace bugs down the road if we do so. I'd
rather pursue Bruce's proposal, as the logic behind it is isolated to
the VM/Runtime level.

There, at least at first sight, is not really a performance hit with his
idea. We're essentially talking about overriding method reference
targets in the VM during compilation. Once method compilation is
complete, whether it is stored through AOT or in a cache after JIT, the
native code will have a direct reference to the implementing plug,
override method, etc. Then it's up to the linker/loader, and whatever
ABI we support (ELF64?), to do the usual symbol table patchups. But
that's another job to do, and outside the scope of what this is trying
to solve. Bruce's plugging is for the CIL level.

--S

tgiphil wrote:
>
> From: tgiphil
>
> Thanks Illuminus. That was well thought out.
>
> Just to explore another idea, would it be more advantageous to merge
> the mscorlib with plugs and produce a patched mscorlib? This way there
> is no plug assembly at runtime. And the patched mscorlib (because of
> the plugs) reference methods and types in the MOSA.Runtime (or others)
> as necessary. Just a thought for discussion (I'm not advocating it).
> -Phil
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCHOR#Post175843>.
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=%5Bmosa:52301%5D>)
>
> To start a new discussion for this project, email
> [email removed] <mailto:[email removed]>
>
> You are receiving this email because you subscribed to this discussion
> on CodePlex. You can unsubscribe or change your settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/mosa> on
> codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any
> posts to this discussion will also be available online at codeplex.com
>
Coordinator
Apr 8, 2009 at 6:08 AM
I personally think patching in the compiler is a bad
idea. Why? The answer is simple: You add an additional
performance penalty for every method call you compile too
lookup what the real target is.

Can't we pursue the path we outlined many moons ago by using
"driver" interfaces inside mono's corlib and pass them patches
to cleanup their code to support it out of the box?

I mean nothing prevents mscorlib itself to forward the call,
which is even being inlined if the field can be analyzed to
target the same object always?

In short: I'm against the idea of plugs and further
complicating the work the compiler does, when there's a
simple (and cleanly testable) approach available.

Mike

> -----Ursprüngliche Nachricht-----
> Von: sbalmos [mailto:notifications@codeplex.com]
> Gesendet: Mittwoch, 8. April 2009 06:26
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: sbalmos
>
> I'd personally shy away now from doing a patched mscorlib. It
> will most likely introduce hard-to-trace bugs down the road
> if we do so. I'd rather pursue Bruce's proposal, as the logic
> behind it is isolated to the VM/Runtime level.
>
> There, at least at first sight, is not really a performance
> hit with his idea. We're essentially talking about overriding
> method reference targets in the VM during compilation. Once
> method compilation is complete, whether it is stored through
> AOT or in a cache after JIT, the native code will have a
> direct reference to the implementing plug, override method,
> etc. Then it's up to the linker/loader, and whatever ABI we
> support (ELF64?), to do the usual symbol table patchups. But
> that's another job to do, and outside the scope of what this
> is trying to solve. Bruce's plugging is for the CIL level.
>
> --S
>
> tgiphil wrote:
> >
> > From: tgiphil
> >
> > Thanks Illuminus. That was well thought out.
> >
> > Just to explore another idea, would it be more advantageous
> to merge
> > the mscorlib with plugs and produce a patched mscorlib?
> This way there
> > is no plug assembly at runtime. And the patched mscorlib
> (because of
> > the plugs) reference methods and types in the MOSA.Runtime
> (or others)
> > as necessary. Just a thought for discussion (I'm not advocating it).
> > -Phil
> >
> > Read the full discussion online
> >
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post175843>.
> >
> > To add a post to this discussion, reply to this email
> ([email removed]
> > <mailto:[email removed]?subject=%5Bmosa:52301%5D>)
> >
> > To start a new discussion for this project, email [email removed]
> > <mailto:[email removed]>
> >
> > You are receiving this email because you subscribed to this
> discussion
> > on CodePlex. You can unsubscribe or change your settings
> >
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on
> > codePlex.com.
> >
> > Please note: Images and attachments will be removed from
> emails. Any
> > posts to this discussion will also be available online at
> codeplex.com
> >
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post176959> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>
Developer
Apr 8, 2009 at 2:50 PM
I started out my reply with some lengthy paragraphs, but I've broken it down to bullet points for easier reading and response:

  • The plug-patching in the compiler would honestly be an additional stage (or more than one), and it could only be active if you've passed the right parameters to the compiler anyway, (if an obvious plug-implementing assembly is passed in and it isn't enabled, I think it would constitute an error condition.)
    • Plug-patching shouldn't be something that the compiler implicitly does all the time - but treated with the same kind of reverence and restrictions as unsafe code.
  • I don't think it is any more or less cleanly testable than when we mainly csc/mcs the corlib with our changes.
  • csc/mcs compiling the corlib ourselves is a bad idea:
    • Mono already maintains corlib for us (not that we would never contribute in that direction)
    • If we compile it ourselves, referencing some other assembly that has the abstraction for our custom implementations, we create a circular assembly reference situation, because that assembly will surely be using corlib too
    • Not to mention we have to be paranoid about maintaining binary compatibility with the real corlib for serialization purposes. (Renamed/added/removed fields in corlib types spells trouble.)
  • Just because I coined the "Plug" term and mentioned Cosmos and SharpOS doesn't mean I want to match their flaws:
    • My approach is uses identical method signatures and field signatures as the corlib type being plugged in each case. No weird constructor plugs to retrieve fields from the target type.
    • My approach tags a plug-implementing type with an attribute that takes the actual target type as a parameter, not a string naming the target namespace
Occasionally we get a flurry of ideas around here. And as someone who didn't write (and can't easily grasp) 95% of the runtime and compiler code around here, I know my opinion doesn't carry much weight. And its hard not knowing what is and isn't planned, so I'll admit part of my enthusiasm is partially just because I know I could literally start hacking some of this stuff together within the next milestone or two, as I've been losing patience and I can't wait around for the master plan to magically happen eventually. Plugging can be used to address multiple problems, and has a small learning curve, which equates to community involvement and lots of flexability.

Slap me if I'm wrong, but I feel very strongly about this...


Developer
Apr 8, 2009 at 3:25 PM
I think there's a disconnect with Bruce wanting this as a compiler
stage. It might be because of how SharpOS used to do their compiler,
specifically the handling of the call CIL opcodes. But nonetheless,
let's take this angle - what Bruce is proposing is functionality that
should be in the runtime/VM, not in the compiler itself. Let's step
through the semantics of how the compiler handles the call opcodes (I
think?):

Let's say Assembly A has Foo.Bar(), which calls mscorlib's
Object.GetHashCode():

- Compiler is compiling Foo.Bar(), comes across a call opcode to
Object.GetHashCode().
- This is where I'm fuzzy: Depending on the type of call opcode, the
compiler can resolve the reference itself (works now), or needs to
delegate the VM/runtime (not implemented). I believe this is dependent
on whether it's a static method or normal/virtual method.
- The not-yet-implemented VM/runtime delegate of the call opcode would
figure out what object instance method to return a reference to, which
the compiler can use to continue on its merry way.

This implementation of the call opcode in the VM/runtime is where I
believe we should put Bruce's overriding logic. It can determine,
ultimately, which implementing plug / override / whatever to return to
the compiler, and then the compiler is none the wiser. The only change
that would have to be made to the compiler's pipeline is the handling of
static calls. I'll leave that to others to debate how to handle that.

Hope that made sense.

--S
Developer
Apr 8, 2009 at 3:45 PM
I actually had not considered that. I'll need time to properly toil over it in my head. Initially it sounds like more trouble to me, but I admit that's from a lack of understanding about the direction the runtime is going in.

I think we need to have some more macro discussions about the runtime (i.e. MOSA.Runtime), how we want to tie into corlib, etc. I really love 90% of what grover has started, but I can't retain enough of it at any one time to really know how to approach it intelligently and productively.

Either way, (doing it in compiler or doing it in runtime), compiler changes need to be made. There are several ways it can be done completely within the compiler, as I was originally pushing at (completely in the linker, like SharpOS, or somewhere as an IR-modifying stage, all with pre-requisite stages that do preparation work).

I'll admit I like having it all in optional compiler stages, to help keep the runtime code clean and open to other approaches, but it definitely needs more thought and discussion.
Coordinator
Apr 8, 2009 at 7:14 PM
An this is where I have my issue with the plugging. You see
the CIL has three call opcodes:

- call
- callvirt
- calli

The interesting point here is that the (Microsoft) C# compiler
always uses the callvirt opcode, even if it could do a
straight call. There's a semantical difference between call/
callvirt: callvirt checks that the this-reference is not null,
if it is null, it implicitly throws an NullReferenceException.

call does not do this and is used only to invoke static methods.
calli can't be emitted by the C# compiler, the only one I'm
aware of that emits it is the MC++ compiler.

Now the issue is where the plugging occurs: If we do this on
every compilation of the call*-opcodes, I'm strongly against
it.

Where we can think about plugging is in our module loader. It
could be made smart enough to check for a *.plugs.dll, everytime
it loads a new module from disk. This replacement mechanism
*must* then override the RuntimeMember-derived types of the
original module. This way the plugging would be completely
transparent to the compiler and only involve the loader.

There remains two problems:

1. Taking the mono-corlib as it is will cause us headaches in
any case, because it will still contain code references
(P/Invoke) which we do not support.

2. Even with plugs moved into a seperate assembly, we'll
still have circular dependencies. And this is going to be
a headache.

Other comments:

And finally, serialization is not an issue. Microsoft has
done a pretty good job there with .NET 2.0 to be able to
resolve compatibility issues at runtime. And even we have
to stick to serialization compatibility with the
Microsoft implementation. To get my point of a merged corlib
better over the wire: I'm not talking about forking the
Mono corlib. I'm talking about pulling certain labelled/
released version from their SVN and plugging it in our
build process. This should be very easy with partial
classes/methods. Our replacements do not have to touch
the mono code, if we can get them to move their
MethodImpl.InternalCall things into seperate files, which
we'll exclude in our MSBuild scripts.

For example if we take String.cs and ask Mono to move
all internal calls into a String.Internal.cs and make both
partial, we could exclude String.Internal.cs in our MSBuild
scripts and instead include String.Mosa.cs - and the
compiler does the jobs for us. It verifies at compilation time
that all required internal calls are still available. We can
verify at build time that the new String class behaves as the
old class. And this is what I mean by improved testability.

Mike


> -----Ursprüngliche Nachricht-----
> Von: sbalmos [mailto:notifications@codeplex.com]
> Gesendet: Mittwoch, 8. April 2009 16:26
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: sbalmos
>
> I think there's a disconnect with Bruce wanting this as a
> compiler stage. It might be because of how SharpOS used to do
> their compiler, specifically the handling of the call CIL
> opcodes. But nonetheless, let's take this angle - what Bruce
> is proposing is functionality that should be in the
> runtime/VM, not in the compiler itself. Let's step through
> the semantics of how the compiler handles the call opcodes (I
> think?):
>
> Let's say Assembly A has Foo.Bar(), which calls mscorlib's
> Object.GetHashCode():
>
> - Compiler is compiling Foo.Bar(), comes across a call opcode
> to Object.GetHashCode().
> - This is where I'm fuzzy: Depending on the type of call
> opcode, the compiler can resolve the reference itself (works
> now), or needs to delegate the VM/runtime (not implemented).
> I believe this is dependent on whether it's a static method
> or normal/virtual method.
> - The not-yet-implemented VM/runtime delegate of the call
> opcode would figure out what object instance method to return
> a reference to, which the compiler can use to continue on its
> merry way.
>
> This implementation of the call opcode in the VM/runtime is
> where I believe we should put Bruce's overriding logic. It
> can determine, ultimately, which implementing plug / override
> / whatever to return to the compiler, and then the compiler
> is none the wiser. The only change that would have to be made
> to the compiler's pipeline is the handling of static calls.
> I'll leave that to others to debate how to handle that.
>
> Hope that made sense.
>
> --S
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post177183> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>
Coordinator
Apr 8, 2009 at 7:17 PM
Bruce,

Even though I have moved the plugging in my last mail into the
loader I still consider it one more point of failure and
vulnerability I don't really like to have.

I still want to merge our plugs with the Mono corlib at C#
compilation time - not our compiler, not our runtime.

Mike

> -----Ursprüngliche Nachricht-----
> Von: illuminus86 [mailto:notifications@codeplex.com]
> Gesendet: Mittwoch, 8. April 2009 16:45
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: illuminus86
>
> I actually had not considered that. I'll need time to
> properly toil over it in my head. Initially it sounds like
> more trouble to me, but I admit that's from a lack of
> understanding about the direction the runtime is going in.
>
> I think we need to have some more macro discussions about the
> runtime (i.e. MOSA.Runtime), how we want to tie into corlib,
> etc. I really love 90% of what grover has started, but I
> can't retain enough of it at any one time to really know how
> to approach it intelligently and productively.
>
> Either way, (doing it in compiler or doing it in runtime),
> compiler changes need to be made. There are several ways it
> can be done completely within the compiler, as I was
> originally pushing at (completely in the linker, like
> SharpOS, or somewhere as an IR-modifying stage, all with
> pre-requisite stages that do preparation work).
>
> I'll admit I like having it all in optional compiler stages,
> to help keep the runtime code clean and open to other
> approaches, but it definitely needs more thought and discussion.
>
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post177191> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>
Developer
Apr 8, 2009 at 9:24 PM
Just for the sake of completeness in the debate, what issues do you have
with doing a per-call lookup of plugs? Maybe I'm simplifying it too much
by thinking it's just a few more lines of code to see if there's an
overriding method defined in special plug assemblies.

Also, to handle PInvokes, InternalCalls, etc, would it be possible for
the call opcode handler in the compiler to see that the method being
called is marked extern, and then see if the plug overrides it that way?
Having Mono split the InternalCalls/PInvokes into a partial class would
be nice, and ultimately would help them to better separate into a HAL.
But I think that would be a while before that would happen.

The circular referencing will just have to be watched with judicious,
controlled coding of the plugs. Most (all?) of the plugs are meant to be
interacting with the VM/Runtime anyway, and that's it. They shouldn't be
calling back into the BCL. We're essentially talking about defining our
libc <-> kernel syscall format here, to oversimplify things.

--S

__grover wrote:
>
> From: __grover
>
> Bruce,
>
> Even though I have moved the plugging in my last mail into the
> loader I still consider it one more point of failure and
> vulnerability I don't really like to have.
>
> I still want to merge our plugs with the Mono corlib at C#
> compilation time - not our compiler, not our runtime.
>
> Mike
>
> > -----Ursprüngliche Nachricht-----
> > Von: illuminus86 [mailto:[email removed]
> > Gesendet: Mittwoch, 8. April 2009 16:45
> > An: [email removed]
> > Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
> >
> > From: illuminus86
> >
> > I actually had not considered that. I'll need time to
> > properly toil over it in my head. Initially it sounds like
> > more trouble to me, but I admit that's from a lack of
> > understanding about the direction the runtime is going in.
> >
> > I think we need to have some more macro discussions about the
> > runtime (i.e. MOSA.Runtime), how we want to tie into corlib,
> > etc. I really love 90% of what grover has started, but I
> > can't retain enough of it at any one time to really know how
> > to approach it intelligently and productively.
> >
> > Either way, (doing it in compiler or doing it in runtime),
> > compiler changes need to be made. There are several ways it
> > can be done completely within the compiler, as I was
> > originally pushing at (completely in the linker, like
> > SharpOS, or somewhere as an IR-modifying stage, all with
> > pre-requisite stages that do preparation work).
> >
> > I'll admit I like having it all in optional compiler stages,
> > to help keep the runtime code clean and open to other
> > approaches, but it definitely needs more thought and discussion.
> >
Coordinator
Apr 8, 2009 at 9:58 PM
The issue I have is the fact that they're done per call, which even in the
case of optimized hashtables is still a penalty.

I remember that we did build a list of methods, which needed plugging in
Mono and it wasn't that large a list. Maybe someone could talk with Miguel
about this in a broader view and contribute patches to Mono for this very
reason to speed up their migration?

About the circular dependency case - lets just do an example - say we're
plugging String.InternalSplit (an actual extern from Mono trunk). It
signature is as follows:

[MethodImplAttribute (MethodImplOptions.InternalCall)]
private extern String[] InternalSplit (char[] separator, int count, int
options);

So what do we have: We need to implement a method, which in order to be
right needs to create *new* instances of two object types: Array, String. A
simple implementation
might look like this:

String[] InternalSplit(char[] seperator, int count, int options)
{
int start = 0, index = 0;
List<string> result = new List<string>();

foreach (char ch in this)
{
if (Array.IndexOf(seperators, ch) != -1)
{
// Split position
string part = this.Substring(start, index - start + 1);
result.Add(part);
start = index + 1;
}

index++;
}

return result.ToArray();
}

A very simple untested implementation from the top of my head. If you look
closely, you can't do a managed implementation of this without:

- At least referencing the String class again (circular dependency #1)
- Using the Array class (circular dependency #2)
- Use System.IEnumerable/CharEnumerator or the indexer from System.String
(this) (circular dependency #3)

This code *works* in a partial environment I suggested earlier. In other
cases we'll have issues during compilation in any case.

The C# partial keyword was made to exactly solve these issues. And it also
solves our problem of doing too much magic in our runtime/compiler.

Mike

> -----Ursprüngliche Nachricht-----
> Von: sbalmos [mailto:notifications@codeplex.com]
> Gesendet: Mittwoch, 8. April 2009 22:25
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: sbalmos
>
> Just for the sake of completeness in the debate, what issues
> do you have with doing a per-call lookup of plugs? Maybe I'm
> simplifying it too much by thinking it's just a few more
> lines of code to see if there's an overriding method defined
> in special plug assemblies.
>
> Also, to handle PInvokes, InternalCalls, etc, would it be
> possible for the call opcode handler in the compiler to see
> that the method being called is marked extern, and then see
> if the plug overrides it that way?
> Having Mono split the InternalCalls/PInvokes into a partial
> class would be nice, and ultimately would help them to better
> separate into a HAL.
> But I think that would be a while before that would happen.
>
> The circular referencing will just have to be watched with
> judicious, controlled coding of the plugs. Most (all?) of the
> plugs are meant to be interacting with the VM/Runtime anyway,
> and that's it. They shouldn't be calling back into the BCL.
> We're essentially talking about defining our libc <-> kernel
> syscall format here, to oversimplify things.
>
> --S
>
> __grover wrote:
> >
> > From: __grover
> >
> > Bruce,
> >
> > Even though I have moved the plugging in my last mail into the
> > loader I still consider it one more point of failure and
> > vulnerability I don't really like to have.
> >
> > I still want to merge our plugs with the Mono corlib at C#
> > compilation time - not our compiler, not our runtime.
> >
> > Mike
> >
> > > -----Ursprüngliche Nachricht-----
> > > Von: illuminus86 [mailto:[email removed]
> > > Gesendet: Mittwoch, 8. April 2009 16:45
> > > An: [email removed]
> > > Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
> > >
> > > From: illuminus86
> > >
> > > I actually had not considered that. I'll need time to
> > > properly toil over it in my head. Initially it sounds like
> > > more trouble to me, but I admit that's from a lack of
> > > understanding about the direction the runtime is going in.
> > >
> > > I think we need to have some more macro discussions about the
> > > runtime (i.e. MOSA.Runtime), how we want to tie into corlib,
> > > etc. I really love 90% of what grover has started, but I
> > > can't retain enough of it at any one time to really know how
> > > to approach it intelligently and productively.
> > >
> > > Either way, (doing it in compiler or doing it in runtime),
> > > compiler changes need to be made. There are several ways it
> > > can be done completely within the compiler, as I was
> > > originally pushing at (completely in the linker, like
> > > SharpOS, or somewhere as an IR-modifying stage, all with
> > > pre-requisite stages that do preparation work).
> > >
> > > I'll admit I like having it all in optional compiler stages,
> > > to help keep the runtime code clean and open to other
> > > approaches, but it definitely needs more thought and discussion.
> > >
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post177339> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>
Coordinator
Apr 8, 2009 at 10:05 PM
Just to move my point further, I tried compiling the example in VS2008:

The first errors I get are:

warning CS1685: The predefined type 'System.String' is defined in multiple
assemblies in the global alias; using definition from
'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll'
D:\test\Project1\String.Mosa.cs(12,17): warning CS0436: The type
'System.String' in 'D:\test\Project1\String.Mosa.cs' conflicts with the
imported type 'string' in
'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll'. Using the type
defined in 'D:\test\Project1\String.Mosa.cs'.
D:\test\Project1\String.Mosa.cs(10,26): (Related location)
c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll: (Related file)

Next I get the following errors in the sample code:

using System;
using System.Collections.Generic;

namespace System
{
public class String
{
String[] InternalSplit(char[] seperator, int count, int options)
{
int start = 0, index = 0;
List<string> result = new List<string>();

// D:\test\Project1\String.Mosa.cs(17,13): error CS1579: foreach statement
cannot operate on variables of type 'System.String' because 'System.String'
does not contain a public definition for 'GetEnumerator'
foreach (char ch in this)
{
if (Array.IndexOf(seperators, ch) != -1)
{
// Split position
// D:\test\Project1\String.Mosa.cs(22,40): error CS1061:
'System.String' does not contain a definition for 'Substring' and no
extension method 'Substring' accepting a first argument of type
'System.String' could be found (are you missing a using directive or an
assembly reference?)
string part = this.Substring(start, index - start + 1);
result.Add(part);
start = index + 1;
}

index++;
}
}
}

// D:\test\Project1\String.Mosa.cs(30,20): error CS0029: Cannot implicitly
convert type 'string[]' to 'System.String[]'
return result.ToArray();
}

I hope the formatting isn't to bad :(

Mike

> -----Ursprüngliche Nachricht-----
> Von: __grover [mailto:notifications@codeplex.com]
> Gesendet: Mittwoch, 8. April 2009 22:59
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: __grover
>
> The issue I have is the fact that they're done per call,
> which even in the case of optimized hashtables is still a penalty.
>
> I remember that we did build a list of methods, which needed
> plugging in Mono and it wasn't that large a list. Maybe
> someone could talk with Miguel about this in a broader view
> and contribute patches to Mono for this very reason to speed
> up their migration?
>
> About the circular dependency case - lets just do an example
> - say we're plugging String.InternalSplit (an actual extern
> from Mono trunk). It signature is as follows:
>
> [MethodImplAttribute (MethodImplOptions.InternalCall)]
> private extern String[] InternalSplit (char[] separator, int
> count, int options);
>
> So what do we have: We need to implement a method, which in
> order to be right needs to create *new* instances of two
> object types: Array, String. A simple implementation might
> look like this:
>
> String[] InternalSplit(char[] seperator, int count, int
> options) { int start = 0, index = 0; List<string> result =
> new List<string>();
>
> foreach (char ch in this)
> {
> if (Array.IndexOf(seperators, ch) != -1) { // Split position
> string part = this.Substring(start, index - start + 1);
> result.Add(part); start = index + 1; }
>
> index++;
> }
>
> return result.ToArray();
> }
>
> A very simple untested implementation from the top of my
> head. If you look closely, you can't do a managed
> implementation of this without:
>
> - At least referencing the String class again (circular dependency #1)
> - Using the Array class (circular dependency #2)
> - Use System.IEnumerable/CharEnumerator or the indexer from
> System.String
> (this) (circular dependency #3)
>
> This code *works* in a partial environment I suggested
> earlier. In other cases we'll have issues during compilation
> in any case.
>
> The C# partial keyword was made to exactly solve these
> issues. And it also solves our problem of doing too much
> magic in our runtime/compiler.
>
> Mike
>
> > -----Ursprüngliche Nachricht-----
> > Von: sbalmos [mailto:[email removed]
> > Gesendet: Mittwoch, 8. April 2009 22:25
> > An: [email removed]
> > Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
> >
> > From: sbalmos
> >
> > Just for the sake of completeness in the debate, what issues do you
> > have with doing a per-call lookup of plugs? Maybe I'm
> simplifying it
> > too much by thinking it's just a few more lines of code to see if
> > there's an overriding method defined in special plug assemblies.
> >
> > Also, to handle PInvokes, InternalCalls, etc, would it be
> possible for
> > the call opcode handler in the compiler to see that the
> method being
> > called is marked extern, and then see if the plug overrides it that
> > way?
> > Having Mono split the InternalCalls/PInvokes into a partial class
> > would be nice, and ultimately would help them to better
> separate into
> > a HAL.
> > But I think that would be a while before that would happen.
> >
> > The circular referencing will just have to be watched with
> judicious,
> > controlled coding of the plugs. Most (all?) of the plugs
> are meant to
> > be interacting with the VM/Runtime anyway, and that's it. They
> > shouldn't be calling back into the BCL.
> > We're essentially talking about defining our libc <->
> kernel syscall
> > format here, to oversimplify things.
> >
> > --S
> >
> > __grover wrote:
> > >
> > > From: __grover
> > >
> > > Bruce,
> > >
> > > Even though I have moved the plugging in my last mail into the
> > > loader I still consider it one more point of failure and
> > > vulnerability I don't really like to have.
> > >
> > > I still want to merge our plugs with the Mono corlib at C#
> > > compilation time - not our compiler, not our runtime.
> > >
> > > Mike
> > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: illuminus86 [mailto:[email removed]
> > > > Gesendet: Mittwoch, 8. April 2009 16:45
> > > > An: [email removed]
> > > > Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
> > > >
> > > > From: illuminus86
> > > >
> > > > I actually had not considered that. I'll need time to properly
> > > > toil over it in my head. Initially it sounds like more
> trouble to
> > > > me, but I admit that's from a lack of understanding about the
> > > > direction the runtime is going in.
> > > >
> > > > I think we need to have some more macro discussions about the
> > > > runtime (i.e. MOSA.Runtime), how we want to tie into
> corlib, etc.
> > > > I really love 90% of what grover has started, but I
> can't retain
> > > > enough of it at any one time to really know how to approach it
> > > > intelligently and productively.
> > > >
> > > > Either way, (doing it in compiler or doing it in runtime),
> > > > compiler changes need to be made. There are several
> ways it can be
> > > > done completely within the compiler, as I was
> originally pushing
> > > > at (completely in the linker, like SharpOS, or somewhere as an
> > > > IR-modifying stage, all with pre-requisite stages that do
> > > > preparation work).
> > > >
> > > > I'll admit I like having it all in optional compiler stages, to
> > > > help keep the runtime code clean and open to other
> approaches, but
> > > > it definitely needs more thought and discussion.
> > > >
> >
> > Read the full discussion online
> > <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
> OR#Post177339> .
> >
> > To add a post to this discussion, reply to this email
> ([email removed]
> > <mailto:[email removed]?subject=[mosa:52301]> )
> >
> > To start a new discussion for this project, email [email removed]
> >
> > You are receiving this email because you subscribed to this
> discussion
> > on CodePlex. You can unsubscribe or change your settings
> > <http://www.codeplex.com/site/discussions/project/unsubscribe/
> > mosa> on codePlex.com.
> >
> > Please note: Images and attachments will be removed from
> > emails. Any posts to this discussion will also be available
> > online at codeplex.com
> >
> >
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post177354> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>
Developer
Apr 9, 2009 at 2:53 AM
Okay, I think this is actually partially why Phil (and to a degree,
SharpOS) was advocating the need for a KCL of sorts, a kernel library.
If you notice in C-based kernels, anything they need, like malloc,
sprintf, and other such utility functions, are all duplicated and
rewritten to be internally-safe. Thus you end up with kmalloc,
ksnprintf, and others.

The partial class idea is good. I'm not sure, though, if that would
create context issues where we're essentially running kernel / VM code
inside the calling "userspace" code's AppDomain. For that matter, is
there any possible issue with exposure of VM / Runtime code in the C#
compiled mscorlib? Something still tells me using
MethodImplAttribute.InternalCall is the "more proper" way to go in the
grand scheme of things.

I was doing some reading of the current MOSA trunk code, and we in fact
already have handling of InternalCall written in. If you look at
AssemblyLinkerStage.IsResolved(), it calls ResolveInternalCall() if the
method being linked is decorated with InternalCall. This, in turn, calls
ITypeSystem.GetImplementationForInternalCall(), which itself looks
through its known types (and also keeps a hashtable cache) that are
decorated with Mosa.Runtime.Vm.InternalCallImplAttribute. From there,
individual methods are decorated with VmCallAttribute and
CompilerSupportAttribute, subclasses of InternalCallImplAttribute.

Sooooo... I'm wondering if the below code is a good simple test case
(I'm still at work at 9:40pm and can't immediately test this, oy...). No
idea really if it compiles, as I'm not quite sure what specific
InternalCallImplAttribute to use. To a degree, I'm following the
skeleton code in CompilerSupport.cs and RuntimeSupport.cs But the idea
is the same.

namespace Test
{
class TestDriver
{
[MethodImplAttribute(MethodImplOptions.InternalCall)]
private static extern void TestInternal();

public static void Main(string[] args)
{
TestInternal();
}
}

[Mosa.Runtime.Vm.InternalCallImplAttribute]
class TestInternal
{
[Mosa.Runtime.Vm.CompilerSupportAttribute]
public void TestInternal()
{
System.Console.WriteLine("It worked!");
}
}
}

Also, Mike, I'm wondering if the compiler was just not liking the fact
that you also named your test "plug" class System.String, and thus you
sort of created your own loop. What happens if your test InternalSplit()
was in, say, Mosa.Vm.Plugs.String?

Just throwing out some options, as it looks as if the handling of
InternalCall was already sort of written in.

--S

__grover wrote:
>
> From: __grover
>
> Just to move my point further, I tried compiling the example in VS2008:
>
> The first errors I get are:
>
> warning CS1685: The predefined type 'System.String' is defined in multiple
> assemblies in the global alias; using definition from
> 'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll'
> D:\test\Project1\String.Mosa.cs(12,17): warning CS0436: The type
> 'System.String' in 'D:\test\Project1\String.Mosa.cs' conflicts with the
> imported type 'string' in
> 'c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll'. Using
> the type
> defined in 'D:\test\Project1\String.Mosa.cs'.
> D:\test\Project1\String.Mosa.cs(10,26): (Related location)
> c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll: (Related file)
>
> Next I get the following errors in the sample code:
>
> using System;
> using System.Collections.Generic;
>
> namespace System
> {
> public class String
> {
> String[] InternalSplit(char[] seperator, int count, int options)
> {
> int start = 0, index = 0;
> List<string> result = new List<string>();
>
> // D:\test\Project1\String.Mosa.cs(17,13): error CS1579: foreach statement
> cannot operate on variables of type 'System.String' because
> 'System.String'
> does not contain a public definition for 'GetEnumerator'
> foreach (char ch in this)
> {
> if (Array.IndexOf(seperators, ch) != -1)
> {
> // Split position
> // D:\test\Project1\String.Mosa.cs(22,40): error CS1061:
> 'System.String' does not contain a definition for 'Substring' and no
> extension method 'Substring' accepting a first argument of type
> 'System.String' could be found (are you missing a using directive or an
> assembly reference?)
> string part = this.Substring(start, index - start + 1);
> result.Add(part);
> start = index + 1;
> }
>
> index++;
> }
> }
> }
>
> // D:\test\Project1\String.Mosa.cs(30,20): error CS0029: Cannot implicitly
> convert type 'string[]' to 'System.String[]'
> return result.ToArray();
> }
>
> I hope the formatting isn't to bad :(
>
> Mike
>
> > -----Ursprüngliche Nachricht-----
> > Von: __grover [mailto:[email removed]
> > Gesendet: Mittwoch, 8. April 2009 22:59
> > An: [email removed]
> > Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
> >
> > From: __grover
> >
> > The issue I have is the fact that they're done per call,
> > which even in the case of optimized hashtables is still a penalty.
> >
> > I remember that we did build a list of methods, which needed
> > plugging in Mono and it wasn't that large a list. Maybe
> > someone could talk with Miguel about this in a broader view
> > and contribute patches to Mono for this very reason to speed
> > up their migration?
> >
> > About the circular dependency case - lets just do an example
> > - say we're plugging String.InternalSplit (an actual extern
> > from Mono trunk). It signature is as follows:
> >
> > [MethodImplAttribute (MethodImplOptions.InternalCall)]
> > private extern String[] InternalSplit (char[] separator, int
> > count, int options);
> >
> > So what do we have: We need to implement a method, which in
> > order to be right needs to create *new* instances of two
> > object types: Array, String. A simple implementation might
> > look like this:
> >
> > String[] InternalSplit(char[] seperator, int count, int
> > options) { int start = 0, index = 0; List<string> result =
> > new List<string>();
> >
> > foreach (char ch in this)
> > {
> > if (Array.IndexOf(seperators, ch) != -1) { // Split position
> > string part = this.Substring(start, index - start + 1);
> > result.Add(part); start = index + 1; }
> >
> > index++;
> > }
> >
> > return result.ToArray();
> > }
> >
> > A very simple untested implementation from the top of my
> > head. If you look closely, you can't do a managed
> > implementation of this without:
> >
> > - At least referencing the String class again (circular dependency #1)
> > - Using the Array class (circular dependency #2)
> > - Use System.IEnumerable/CharEnumerator or the indexer from
> > System.String
> > (this) (circular dependency #3)
> >
> > This code *works* in a partial environment I suggested
> > earlier. In other cases we'll have issues during compilation
> > in any case.
> >
> > The C# partial keyword was made to exactly solve these
> > issues. And it also solves our problem of doing too much
> > magic in our runtime/compiler.
> >
> > Mike
> >
Developer
Apr 9, 2009 at 6:02 AM
[Afterthought: Sorry, lots of caffeine!]

A few notes, merely as an intelligent continuation of the conversation we are having, in response to above raised points:

  • For better or worse, my methodology does not prevent the overriding of non-extern methods. My methodology also effectively imports methods you aren't overriding into the class implementing the plugs. It also imports fields. (Giving you full access to the surrounding class.)
    --But it still doesn't equate to anything more than inserting the method body as the method body of the original method being called.
    • The compiler performance can be fairly optimized by making sure that as the compiler digs into the types and then methods of what is being compiled, it can concisely refine where it is searching for possible methods that are overriding the current one. With override-mapping as a specific optional part of compilation, I think that with the awesome architecture you've put in place that 99% we won't have any override.
  • My PlugGen tool also attempts to map referenced types into the BCL plug-implementing skeleton. So that plug-implementation methods always have a way to fully simulate the other corlib types being interacted with in an intimate way. ('Internal' scoped members.) (My tool doesn't scan method bodies, but it is not hard to dump a skeleton of a type you are interested in.) Remember that all fields, and no methods that you are implementing, due to the way my attributes are done, would be effectively ignored by our compiler. They are available so that C# compiler can handle the assembly, but always effectively map to something we aren't responsible for. The only caveat is we'd need a MOSA-compiler intrinsic to sometimes convert between a corlib-context type to a plug-assembly context type. The intrinsic wouldn't do anything, other than satisfying method signatures or dealing (possibly) with value types. (The intrinsic could totally be of signature Func<T1,T2>.)
  • My methodology honestly treats all methods the same. (Even methods most .NET developers don't think of as methods, like property set/get, and event property add/remove/call.) Plus pinvokes and icalls. The fact that there is marshalling information provided by annotations in corlib for pinvokes doesn't have to be relevant if there is a plug for the method.
  • I specifically, with my tool, output with non-standard namespace names. ("InternalSystem.*", etc.) The mapping of a plug-implementing type with its target is using the TypeOverrideAttribute with the target type itself, (not the type name), as a parameter. Hence the above mentioned need of a compiler intrinsic in order to locally mask or unmask a true corlib type. If global namespace specifiers are used when true references to a corlib type are needed, there are rarely namespace clashes. (But if there are, you can just prefix generated namespace components and then prefix type names.)
  • My tool dumps, I think, 85 classes, from corlib. Not all of which with icalls. Some of which are ancestor classes, generic type constraint referenced classes, or method signature referenced classes, from types/methods that have/are icalls. I don't currently recurse, but I believe it is good coverage and a good represenation of what we would have to deal with regardless as to how we want to implement (plug/no-plug) parts of the corlib.
  • I guess we can't avoid circular dependencies. I guess I really feel like the schema I am proposing doesn't allow you to stray enough to think of a plug-implementing assembly as its own standalone entity. Honestly I see it as a way to reference MOSA.Runtime (minus a few of its current elements that would be redundant of what I am proposing, like the fill-in for System.Object) and the kernel handle (aka "ObjectManager" from a few months back) -- without actually making the original Mono BCL actually have to reference the runtime.  Its a way of working with Mono without imposing on Mono.
I guess there isn't a problem with method implementations using parts of the BCL again. The only parts that can't be endlessly circular is the type metadata, threading, and memory allocation. If the first is included in the executable image by the compiler/linker, the second and third are setup by a runtime-constricted initial part of the kernel, we should be good. Even though our runtime metadata tables will be expressed as our own classes (which inherits System.Object which has a reference back to the metadata tables), I think for the purpose of statically linking in the metadata we can have the compiler make sure we don't "serialize" any fields of System.Object. (But any fields up the inheritance chain to that point.)

The circular referencing will just have to be watched with judicious,
controlled coding of the plugs. Most (all?) of the plugs are meant to be
interacting with the VM/Runtime anyway, and that's it. They shouldn't be
calling back into the BCL. We're essentially talking about defining our
libc <-> kernel syscall format here, to oversimplify things.

Agreed.

As far as partial methods, as much as I hate the thought of csc-ing Mono's corlib, it is definitely feasible. I just think we'd have to do all the partial-class creating work, and hope they trunk it. I wouldn't expect much support from the Mono community until we are at least as popular as an unpopular *nix flavor, or we have corporate funds to wave around.

And I like to think of "plug"ing as an opportunity for extensibility, not vulnerability. It gives us a clear path to always being able to easily fill in the gaps for any assembly that uses icall or pinvoke craziness. And just like an unavoidable need to occasionally use 'unsafe' code, I think it is easily enough solved by OS-level code-trusting policy, as a sibling to said issue.

As far as a point of failure, I can't argue much on that point. But I would rather it break in our compiler, than our runtime tables or in Mono's corlib.

Testability is not affected, as far as I can tell, by this methodology. The Mono unit tests would verify the classes behavoirs just as well without or without plugs.

I hope you understand that I picture plugging as a schema, not as a hack. Whether it is implemented in our compiler, in our runtime, both, or neither, I picture it as a standard that can be implemented (and needs to be further fleshed-out), not as an implementation detail itself.

Honestly, a Cecil-based tool to merge an assembly with it's plug assembly wouldn't be too hard to write, other than the issues around tying the plugs to MOSA.Runtime that I have already been pondering. Hrrrm...

Apr 9, 2009 at 9:07 AM
Just to throw a spanner in the works. How about appropriating P/Invoke?

For example:

class String
{
   [DllImport("Mosa.String, %CURRENTPLATFORMASSEMBLY%")] // Yes, compiler magic, but far less.
   private static extern void SplitInternal (ref string source, int delimiter, out string dest);
}

In Mosa.Platform.X86

class String
{
   public static void SplitInternal(void* source, int delimiter, void* dest)
   {
     // We know the struct layout of a string, as such we can work with it using pointers.
     // No cyclic ref!
   }
}

Developer
Apr 9, 2009 at 2:07 PM
Just to throw a spanner in the works. How about appropriating P/Invoke?

Good lord son - you're smokin more crack than I am! But it really doesn't say anything more than any other attribute would. And in fact, I would say that if we didn't do the compile-corlib-with- partial-classes approach that grover has suggested, we would probably want something with a slightly stronger contract. Strings as an attribute parameter, when you could use type references (even if just a type reference to an interface that any assembly could be used to implement, picked at compile time), should only be a last resort. MethodImplAttribute doesn't provide enough information, (other than the current method and type, as the compiler would already know), and DllImportAttribute has a different kind of meaning to other compilers than our own, I really think that if we called out from a compiled assembly like this, MethodImplAttribute is a better fit. And it is already where we need it.

The partial class idea is good. I'm not sure, though, if that would
create context issues where we're essentially running kernel / VM code
inside the calling "userspace" code's AppDomain.

I'm not sure that this is a problem. When a process is set up, according to grover's process design, (and any MOSA OS like Ensemble or SharpOS could use a similar methodology, but implemented with their own special touches), each process get's it's own "ObjectManager", (which I tend to think of more as a "kernel proxy"). Calls to the scheduler for more threads, calls to the VFS, etc. are all marshalled across IPC boundaries (and potentially filtered by the local ObjectManager if special redirects or constrictions are setup for the process). If the code going on is actually indeed running in the kernel, you would still be using "ObjectManager", it just wouldn't be using IPC to talk to the kernel subsystems, because the kernel-local implementation would know where to look.

Okay, I think this is actually partially why Phil (and to a degree,
SharpOS) was advocating the need for a KCL of sorts, a kernel library.
If you notice in C-based kernels, anything they need, like malloc,
sprintf, and other such utility functions, are all duplicated and
rewritten to be internally-safe. Thus you end up with kmalloc,
ksnprintf, and others.

I would have to study/discuss this more to better understand it. The only thing I can imagine honestly differing between the kernel and userspace is honestly just the issue of what would differ between before the kernel's subsystems have started and after. Creating a FileStream before you have a VFS would get you an exception from the ObjectManager. Creating a thread before you have a scheduler would get you an exception from the ObjectManager. Operations that allocate memory before you have a VFS...

...would be a problem we would have to deal with in a different manner. More at the runtime level in general. That's why I think we would setup an initial MOSA.Runtime.RuntimeBase instance at compile time, that is replaced after the kernel gets started. We could go the easy route out, and have some static space in the kernel image that would be used at runtime for the types of allocations that would need to be made before true memory management is off the ground. Or we could go a harder route, and find a way to make memory allocations at runtime (during these initial stages) actually fail, and rely on compiler intrinsics and other forms of compiler magic to have well-known constructs allocated statically at compile time.

The biggest issue with this approach that I initially see is the lack of threading in order to do garbage collection during initial kernel stages. But maybe the lack of ability to do a proper collect until after the scheduler is setup, is a reasonable concession. All of the other concerns are Runtime/ObjectManager concerns, and I think can be handled gracefully.

I was doing some reading of the current MOSA trunk code, and we in fact already have handling of InternalCall written in. If you look at AssemblyLinkerStage.IsResolved(), it calls ResolveInternalCall() if the...

This scares me. It scares me because I now remember reading it, and being curious about playing with it, And I totally forgot it was there. I have no retention when it comes to the MOSA codebase... am I just getting older?
Coordinator
Apr 9, 2009 at 8:25 PM
Hi everyone,

1. Please drop the idea quickly of reusing the P/Invoke mechanism for a
plugging system. Essentially this would mean touch Mono source code even
more than I'm suggesting that we should ;)

2. Yes, MOSA already has InternalCall support with hot-patching and it
basically does what illuminus has been discussing, the only thing it lacks
is loading the plug assembly automagically alongside the main assembly. But
from what I wrote you should have gotten the idea that I dislike what I
wrote there. The reasons were explained previously.

3. In MOSA there's no difference between Kernel and Usermode. I'd personally
like to forget these terms, unless there's a real reason to use the
processor mode rings.

4. illuminus is right about the ObjectManager transfer, essentially it'll
use a special serialization in memory to pass requests to/from the
corresponding IPC server.

5. The scheduler is the very first thing, which will start alongside the
memory management - there's no reason not to design parallel system startup
from the beginning. However I'd personally like to put some IoC sugar into
the mix.

6. What has previously been discussed under the term KCL should actually be
the IPC serialization format to allow multiple (different) corlib versions
to exchange/cross-reference their objects.

Mike

> -----Ursprüngliche Nachricht-----
> Von: illuminus86 [mailto:notifications@codeplex.com]
> Gesendet: Donnerstag, 9. April 2009 15:08
> An: michael.ruck@michaelruck.de
> Betreff: Re: MOSA Runtime, Kernel, and Plugging [mosa:52301]
>
> From: illuminus86
>
> Just to throw a spanner in the works. How about
> appropriating P/Invoke?
>
>
> Good lord son - you're smokin more crack than I am! But it
> really doesn't say anything more than any other attribute
> would. And in fact, I would say that if we didn't do the
> compile-corlib-with- partial-classes approach that grover has
> suggested, we would probably want something with a slightly
> stronger contract. Strings as an attribute parameter, when
> you could use type references (even if just a type reference
> to an interface that any assembly could be used to implement,
> picked at compile time), should only be a last resort.
> MethodImplAttribute doesn't provide enough information,
> (other than the current method and type, as the compiler
> would already know), and DllImportAttribute has a different
> kind of meaning to other compilers than our own, I really
> think that if we called out from a compiled assembly like
> this, MethodImplAttribute is a better fit. And it is already
> where we need it.
>
>
>
> The partial class idea is good. I'm not sure, though,
> if that would
> create context issues where we're essentially running
> kernel / VM code
> inside the calling "userspace" code's AppDomain.
>
>
>
> I'm not sure that this is a problem. When a process is set
> up, according to grover's process design, (and any MOSA OS
> like Ensemble or SharpOS could use a similar methodology, but
> implemented with their own special touches), each process
> get's it's own "ObjectManager", (which I tend to think of
> more as a "kernel proxy"). Calls to the scheduler for more
> threads, calls to the VFS, etc. are all marshalled across IPC
> boundaries (and potentially filtered by the local
> ObjectManager if special redirects or constrictions are setup
> for the process). If the code going on is actually indeed
> running in the kernel, you would still be using
> "ObjectManager", it just wouldn't be using IPC to talk to the
> kernel subsystems, because the kernel-local implementation
> would know where to look.
>
>
>
> Okay, I think this is actually partially why Phil (and
> to a degree,
> SharpOS) was advocating the need for a KCL of sorts, a
> kernel library.
> If you notice in C-based kernels, anything they need,
> like malloc,
> sprintf, and other such utility functions, are all
> duplicated and
> rewritten to be internally-safe. Thus you end up with kmalloc,
> ksnprintf, and others.
>
>
>
> I would have to study/discuss this more to better understand
> it. The only thing I can imagine honestly differing between
> the kernel and userspace is honestly just the issue of what
> would differ between before the kernel's subsystems have
> started and after. Creating a FileStream before you have a
> VFS would get you an exception from the ObjectManager.
> Creating a thread before you have a scheduler would get you
> an exception from the ObjectManager. Operations that allocate
> memory before you have a VFS...
>
> ...would be a problem we would have to deal with in a
> different manner. More at the runtime level in general.
> That's why I think we would setup an initial
> MOSA.Runtime.RuntimeBase instance at compile time, that is
> replaced after the kernel gets started. We could go the easy
> route out, and have some static space in the kernel image
> that would be used at runtime for the types of allocations
> that would need to be made before true memory management is
> off the ground. Or we could go a harder route, and find a way
> to make memory allocations at runtime (during these initial
> stages) actually fail, and rely on compiler intrinsics and
> other forms of compiler magic to have well-known constructs
> allocated statically at compile time.
>
> The biggest issue with this approach that I initially see is
> the lack of threading in order to do garbage collection
> during initial kernel stages. But maybe the lack of ability
> to do a proper collect until after the scheduler is setup, is
> a reasonable concession. All of the other concerns are
> Runtime/ObjectManager concerns, and I think can be handled gracefully.
>
>
>
> I was doing some reading of the current MOSA trunk
> code, and we in fact already have handling of InternalCall
> written in. If you look at AssemblyLinkerStage.IsResolved(),
> it calls ResolveInternalCall() if the...
>
>
>
> This scares me. It scares me because I now remember reading
> it, and being curious about playing with it, And I totally
> forgot it was there. I have no retention when it comes to the
> MOSA codebase... am I just getting older?
>
>
> Read the full discussion online
> <http://mosa.codeplex.com/Thread/View.aspx?ThreadId=52301&ANCH
OR#Post177573> .
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[mosa:52301]> )
>
> To start a new discussion for this project, email
> [email removed]
>
> You are receiving this email because you subscribed to this
> discussion on CodePlex. You can unsubscribe or change your
> settings
> <http://www.codeplex.com/site/discussions/project/unsubscribe/
> mosa> on codePlex.com.
>
> Please note: Images and attachments will be removed from
> emails. Any posts to this discussion will also be available
> online at codeplex.com
>
>