Implement Multithreading #1610

valentinbreiz · 2020-12-27T18:36:56Z

This pull request is not to merge in master yet, this is more an opening for discussion on the implementation of multithreading in Cosmos.

All the original code of the first commit comes from @Og-Rok's work from Aura's fork of Cosmos. (the original pull request can be found here if needed: https://github.com/aura-systems/Cosmos/pull/40)

Here is a short demonstration video: https://www.youtube.com/watch?v=rea5sAkYyec

To do:

fix PC Speaker and process scheduler conflict due to PIT
alloc from new heap system
add test kernel

valentinbreiz · 2020-12-27T18:58:26Z

source/Cosmos.Core/Processing/ProcessorScheduler.cs

+            ProcessContext.m_CurrentContext = context;
+
+            IOPort counter0 = new IOPort(0x40);
+            IOPort cmd = new IOPort(0x43);


Find a way to use another port or something else than Pit (APIC timer?)
With this using PC Speaker won't be possible and will hang the processor scheduler due to PIT conflicts

maybe use PIT implemented in Cosmos, not IOPort PIT?

@AsertCreator they are one in the same

maybe you can implement the nanosecond time in RTC, and use RTC to approximate? or maybe (pure speculation, as a junior dev who knows nothing about low level) reimplement PIT wait to tally up the number of times IRQ 0 has been triggered since the os was powered on, and use this to wait x amount of time. in the case of the second option, it would be possible to implement a system where multiple things can use PIT wait at the same time. see osdev wiki.

RTC is too slow for scheduling and idk if we could make interrupts with that

In fact RTC may not be a bad idea until we have HPET or local APIC timers, it indeed can generate IRQs. I'll try, I hope this won't mess up date time

01:35:10.769067 Msg: Text from kernel: SwitchTask 01:35:10.787021 Msg: Text from kernel: SwitchTask 01:35:10.802028 Msg: Text from kernel: SwitchTask 01:35:10.818118 Msg: Text from kernel: SwitchTask 01:35:10.834006 Msg: Text from kernel: SwitchTask 01:35:10.849588 Msg: Text from kernel: SwitchTask 01:35:10.865546 Msg: Text from kernel: SwitchTask

I can get no more than 60Hz but it's possible

Does that function as a stopgap until APIC gets finished?

Oh. I just looked at the commit and it appears y'all got the APIC timer working. Ok then.

madelynwith5ns · 2020-12-28T02:04:54Z

This would be very beneficial.

Nik300 · 2021-01-03T21:15:57Z

Guys have a huge problem with while loops...
this is the code I am threading... and it just stops after some seconds

public void TaskRun()
{
            while (true)
                Sys.Emulation.Threading.TaskManager.Next();
}

public static void Next()
{
    foreach (Thread thread in LoadedThreads)
    {
       DateTime startTime = DateTime.Now;
       DateTime lastTime = startTime;
       while (lastTime.Second < startTime.Second + 1)
       {
            thread.exec.NextInstruction();
            lastTime = DateTime.Now;
        }
    }
}

Nik300

Really impressive changes here... but have some serious troubles with while loops!
I'll try to track down the error then publish my results

Nik300 · 2021-01-04T18:20:42Z

UPDATE:
Ok so I just found out that the problem isn't the while loop but the keyboard instead...
Here's the thing:
When I load a task as long as I don't press any key it works... but as soon as I click a key the entire system shuts down

🤔

I'll dig more into that

EDIT: I think i may know the problem

The Console.ReadLine is being split across the threads needing it...
let's say that i have 'Thread1' and 'Thread2' and both use 'Console.ReadLine()':
the first thread starts and uses the keyboard to read the first 2 chars, then passes control to the second thread that takes the next two chars, and this happens until the user clicks the ENTER key and one of the two processes handles the result from the keyboard...

Now let's imagine a scenario where one of the two threads doesn't use 'Console.ReadLine()':
as soon as thread1 passes control to thread2 the system crashes, because the thread finds something in its stack that shouldn't be there!

The only way to fix this is by modifying the behaviour of Console.ReadLine

EDIT:
here's the proof:

charlesbetros · 2021-01-04T19:25:28Z

Is it any interrupt or just keyboard?

…

On Jan 4, 2021, at 12:21 PM, F4lc0131 ***@***.***> wrote: UPDATE: Ok so I just found out that the problem isn't the while loop but the keyboard instead... Here's the thing: When I load a task as long as I don't press any key it works... but as soon as I click a key the entire system shuts down 🤔 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Og-Rok · 2021-01-04T20:24:00Z

From what i remember with this, a lot of the core kernel systems (interrupts, heap, device management) are not thread safe, meaning that you will get a lot of situations like this where threads are tripping each other up. A fix for things like the heap would be to implement something like spinlocks

Nik300 · 2021-01-04T20:33:52Z

Is it any interrupt or just keyboard?
…
On Jan 4, 2021, at 12:21 PM, F4lc0131 @.***> wrote: UPDATE: Ok so I just found out that the problem isn't the while loop but the keyboard instead... Here's the thing: When I load a task as long as I don't press any key it works... but as soon as I click a key the entire system shuts down 🤔 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

as of now just the keyboard... I'll try to bust and fix those errors before releasing this to real cosmos

Nik300 · 2021-01-04T21:16:57Z

Actually guys I had an idea on how we could handle this but I'll have to work on a "lock" system, so that a thread can acquire the lock over something (in this case the Console.ReadLine) tell me what you guys think (especially @valentinbreiz)

Nik300 · 2021-01-04T21:53:13Z

For now this should do the job... but i'd like to do something more flexible

UPDATE:

Good news: it doesn't crash!
Bad news: after flushing the first thread, the input just hangs... 😢
have to work on this

UPDATE:

Got it working guys! I'm gonna pull request the code to @valentinbreiz 's fork so that he can port it to this pull request

Nik300 · 2021-01-04T23:54:40Z

UPDATE:
Lol I just realized that there's something weirder going on...
ok so with the Console.ReadLine() fix I still have issues with loading more than one thread that use a while loop...
have to track that down too and fix that :(

UPDATE:
No, i was just being an idiot :D, just forgot to fix the main problem... going to compile and see if it works, but i'll go to bed either if it does/doesn't work

EDIT:
Nope doesn't work, ok i'll work on that tomorrow (i want to point out that as long as the second thread uses Console.ReadLine too, it works... just crashes when the second thread doesn't use it)

UPDATE:
Ok so I think I got another problem like that, and surprisingly the problem is with Console.WriteLine().

Sys.Thread(() => { while (true) Console.WriteLine("test"); });

that's the thread, and happens the same exact thing... shell I add a locksystem to WriteLine too? Or just use the same lock in common with ReadLine?? lemme know

Nik300 · 2021-01-05T12:47:19Z

UPDATE:

Fixed! I think that Cosmos.Clear() needs the same treatment... because as soon as I use the command "cls" the system shuts down

madelynwith5ns · 2021-01-06T04:33:53Z

With the trend we're seeing with methods like Console.ReadLine() Console.WriteLine() and Console.Clear() I'm guessing that there is other methods with this issue.

valentinbreiz · 2021-01-06T20:14:40Z

You're right @TFTWPhoenix, some parts of Cosmos are not thread safe and maybe needs some locks too
But WriteLine or Clear are working fine

github-actions · 2021-04-26T01:56:37Z

Stale pull request message

ThomasBeHappy · 2021-09-06T08:25:03Z

Will this ever be finished? Kinda really waiting for this.

Kiirx · 2022-05-29T12:20:23Z

Any news?

AsertCreator · 2023-03-19T23:00:52Z

i got this to work (maybe). currently there are two problems: stack corruption detection and Thread.Sleep method. first one thinks that stack was corrupted, although it just got replaced. second one is not implemented and makes thread sleep indefinitely, in other words, it makes thread die. when these are not used, multithreading somehow and somewhat works. do i need to submit PR?

zarlo · 2023-03-20T02:42:43Z

i would make a PR as the owner of the RP does not have time to work on this

AsertCreator · 2023-03-20T04:23:02Z

ok

zarlo · 2024-01-07T22:55:36Z

source/Cosmos.Core/Processing/Mutex.cs

+
+        public void Lock()
+        {
+            while (gate != 0) { }


we should yield here so we dont waste as much cpu time

Neil-Ciciglycerin · 2024-01-17T02:30:30Z

i found a bug with how multithreading interacts with the AC97 driver here

Neil-Ciciglycerin · 2024-01-23T17:02:40Z

i found a bug with graphics. i have some system init code in a function called __sysstartup__. there's a null reference exception when init-ing canvas, on main thread or other thread.

valentinbreiz · 2024-01-23T18:29:55Z

i found a bug with graphics. i have some system init code in a function called __sysstartup__. there's a null reference exception when init-ing canvas, on main thread or other thread.

Hello @Neil-Ciciglycerin I think this issue is also in the main branch unfortunally. Could you try and tell me please?

quajak · 2024-01-24T00:25:01Z

source/Cosmos.Core/MMIO.cs

+    /// <summary>
+    /// MMIOBase abstract class.
+    /// </summary>
+    public unsafe abstract class MMIOBase


How does this differ from a MemoryBlock?

quajak · 2024-01-24T00:25:17Z

source/Cosmos.Core/ACPI.cs

                }
            }
+        }
+
+        /*


Do we need this commented out code?

quajak · 2024-01-24T00:28:40Z

source/Cosmos.Core/Memory/Heap.cs

@@ -20,6 +21,8 @@ public enum ObjectGCStatus : ushort
    public static unsafe class Heap
    {
        private static uint* StackStart;
+        private static Mutex mMemeoryGate = new Mutex();


quajak · 2024-01-24T00:29:10Z

source/Cosmos.Core/Memory/Heap.cs

@@ -44,6 +47,11 @@ public static unsafe void Init()
 		/// <returns>New pointer with specified size while maintaining old data.</returns>
        public static byte* Realloc(byte* aPtr, uint newSize)
 		{
+            if (mMemeoryGate != null)


How can the mMemoryGate ever be null? Seems like we are doing useless checks

we could replace it with mMemeoryGate?.Lock(); it would do that same

I would prefer if we dont do needless checks to reduce the performance cost, as these calls happen quite often

quajak · 2024-01-24T00:31:02Z

source/Cosmos.Core/ObjUtilities.cs

+        }
+
+        [PlugMethod(PlugRequired = true)]
+        public static uint GetPointer(Object aVal) { return 0; }


We already have

Cosmos/source/Cosmos.Core/GCImplementation.cs

Line 108 in f7fd569

public static unsafe uint* GetPointer(object aObj) => throw null; // this is plugged

which does more or less the same?
If you need a uint we also have

Cosmos/source/Cosmos.Core/GCImplementation.cs

Line 115 in f7fd569

public static unsafe uint GetSafePointer(object aObj)

quajak · 2024-01-24T00:32:46Z

source/Cosmos.Core_Asm/CPU/CPUUpdateIDTAsm.cs

-
+                else
+                {
+                    new LiteralAssemblerCode("pushad");


Why not use the XS equivalent commands?

quajak · 2024-01-24T00:33:54Z

source/Cosmos.Core_Asm/ObjUtilitiesImpl.cs

+    [Plug(Target = typeof(ObjUtilities))]
+    public static unsafe class ObjUtilitiesImpl
+    {
+        [PlugMethod(Assembler = typeof(ObjUtilitiesGetPointer))]


See previous comment if we really need this method new here.

quajak · 2024-01-24T00:34:16Z

source/Cosmos.Core_Plugs/System/Threading/ThreadImpl.cs

+
+        public static void Ctor(ThreadStart aThis, ThreadStart aEntry)
+        {
+            Console.WriteLine("Thread started");


I think this should be removed.

quajak · 2024-01-24T00:34:32Z

source/Cosmos.Core_Plugs/System/Threading/ThreadImpl.cs

+            Console.WriteLine("Thread started");
+        }
+
+        //    public static void SleepInternal(int ms)


Can this code be removed?

quajak · 2024-01-24T00:35:13Z

source/Cosmos.HAL2/Global.cs


            Console.WriteLine("Starting ACPI");
            debugger.Send("ACPI Init");
            ACPI.Start();

+            Console.WriteLine("Finding PCI Devices");


I don't think we should write as much to the console, since the user cant control this behavior.

Neil-Ciciglycerin · 2024-01-24T02:28:11Z

The last time I checked it wasn't in the main branch, although I was using userkit. I'll check now

Neil-Ciciglycerin · 2024-01-24T15:23:27Z

Yeah, it's in the main branch. I'll go start an issue.

Add changes from @Og-Rok

6bef4e1

valentinbreiz added Area: Kernel Complexity: High Up for Grabs labels Dec 27, 2020

valentinbreiz commented Dec 27, 2020

View reviewed changes

valentinbreiz mentioned this pull request Dec 27, 2020

Multithreading aura-systems/Aura-Operating-System#185

Open

Nik300 suggested changes Jan 4, 2021

View reviewed changes

valentinbreiz added the Work in progress label Jan 25, 2021

github-actions bot added no-pr-activity and removed no-pr-activity labels Apr 26, 2021

MishaTy mentioned this pull request May 14, 2021

plug for System.Threading? #1778

Closed

ThomasBeHappy mentioned this pull request Sep 7, 2021

Implement Nik's Multithreading #1921

Closed

valentinbreiz added 3 commits June 6, 2022 20:00

Merge master.

63b4fcc

Merge branch 'master' into impl-multitask

e9e32aa

Fix conflicts + Lock memory gate using new memory manager

f2716cf

Merge branch 'master' into impl-multitask

e48cda5

This comment was marked as outdated.

Sign in to view

valentinbreiz added 6 commits December 26, 2023 10:45

Merge branch 'master' into impl-multitask

ca9bf96

🐛 Fix memory manager with multithreading

68dc3e0

🐛 Fix IO ports call (now static call)

0c5fdfd

🐛 Fix build

66fe85b

🐛 Fix build (wrong mStackContext path)

d36bfd4

Merge branch 'master' into impl-multitask

ebe5302

zarlo reviewed Jan 7, 2024

View reviewed changes

valentinbreiz added 6 commits January 17, 2024 11:49

Merge branch 'master' into impl-multitask

7eec507

✨ Use local APIC timer instead of PIT for scheduler

8ae051b

Merge remote-tracking branch 'origin/fix/bochsClock' into impl-multitask

5eb0fe7

🐛 Fix scheduler with local apic timer (now works)

fd497d1

Merge branch 'master' into impl-multitask

78f9e08

⚡️ Set 100Hz instead of 1000Hz

72c1436

valentinbreiz requested a review from zarlo January 23, 2024 11:16

zarlo approved these changes Jan 23, 2024

View reviewed changes

zarlo enabled auto-merge January 23, 2024 20:04

quajak requested changes Jan 24, 2024

View reviewed changes

MishaProductions and others added 3 commits January 24, 2024 16:15

Merge branch 'master' into impl-multitask

af2a785

Merge branch 'master' into impl-multitask

485c468

Merge branch 'master' into impl-multitask

106ec6f

                               }
                           }
+                      }
+                      /*

Implement Multithreading #1610

Are you sure you want to change the base?

Implement Multithreading #1610

Conversation

valentinbreiz commented Dec 27, 2020 • edited

valentinbreiz Dec 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valentinbreiz Jan 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

madelynwith5ns commented Dec 28, 2020

Nik300 commented Jan 3, 2021 • edited

Nik300 left a comment

Choose a reason for hiding this comment

Nik300 commented Jan 4, 2021 • edited

charlesbetros commented Jan 4, 2021 via email

Og-Rok commented Jan 4, 2021

Nik300 commented Jan 4, 2021

Nik300 commented Jan 4, 2021

Nik300 commented Jan 4, 2021 • edited

Nik300 commented Jan 4, 2021 • edited

Nik300 commented Jan 5, 2021

madelynwith5ns commented Jan 6, 2021

valentinbreiz commented Jan 6, 2021 • edited

github-actions bot commented Apr 26, 2021

ThomasBeHappy commented Sep 6, 2021

Kiirx commented May 29, 2022

AsertCreator commented Mar 19, 2023

zarlo commented Mar 20, 2023

AsertCreator commented Mar 20, 2023

This comment was marked as outdated.

Choose a reason for hiding this comment

Neil-Ciciglycerin commented Jan 17, 2024 • edited

Neil-Ciciglycerin commented Jan 23, 2024 • edited

valentinbreiz commented Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Neil-Ciciglycerin commented Jan 24, 2024 • edited

Neil-Ciciglycerin commented Jan 24, 2024

valentinbreiz commented Dec 27, 2020 •

edited

valentinbreiz Dec 27, 2020 •

edited

valentinbreiz Jan 17, 2024 •

edited

Nik300 commented Jan 3, 2021 •

edited

Nik300 commented Jan 4, 2021 •

edited

Nik300 commented Jan 4, 2021 •

edited

Nik300 commented Jan 4, 2021 •

edited

valentinbreiz commented Jan 6, 2021 •

edited

Neil-Ciciglycerin commented Jan 17, 2024 •

edited

Neil-Ciciglycerin commented Jan 23, 2024 •

edited

Neil-Ciciglycerin commented Jan 24, 2024 •

edited