You might think I’m crazy, but I’m into this thing “Write your own OS”. I have some commercial ideas on how and why, but I’m willing to share my knowledge of what I’ve found so far. This is the first one in the series, as I’m currently researching stuff and wanted to write them down for others like me.
For starters, it’s more about running your own code directly on CPU, once the PC is turned on, without any underlying OS, than writing OS itself. But the grand master plan still remains – to create my own OS.
The Plan
As for every project, we need a plan on what we’re trying to achieve. My plan is:
- Target modern architecture – I’m going to target Intel64 (I’ve my eye on Ivy Bridge, 3rd gen Core i5 CPU with integrated Intel HD 4000 GPU and B75 Express chipset);
- Avoid legacy, drop it as fast as you can. You can even skip the Protected mode and jump directly to Long mode;
- Jump to C as soon as possible – I’m afraid of Assembly language 🙂
- Forget old interfaces like PCI, IDE, PS/2, Serial/Parallel ports. Why does every tutorial still use such an ancient device as Floppy? And even if my plan comes together – forget Basic Input Output System (BIOS), long live Unified Extensible Firmware Interface (UEFI). For now i’m targeting only modern interfaces – PCIe, SATA, USB;
- Avoid the use of GRUB or any other multiboot bootloader – make my own and allow only my own OS on the system – yeah that’s ego centric decision, but I’m going to use CPU for my self and this blog is all about getting from power button directly to my C code executed as the only code on the CPU.
Further plans for my OS – these considerations will not be covered here, but I’m writing them down, just to show you some crazy ideas of mine:
- Use C, C++ and some popular scripting language (Currently I have my eyes on Python or interpreted C). C for kernel and core routines, C++ for GUI and other stuff that’s easier to use with object-oriented abstraction. Scripting language for user space, so that you could write apps easier, without compilation, with deep integration in to OS;
- URI based system with schema drivers. For example file://localhost/my_documents/ or just /my_documents/ in the file browser;
- No Unix directory tree. I hate it – it does not say anything to a user – when I first started using Linux I didn’t know that /etc is to store configuration files, and I still don’t understand the meaning and differences between /bin, /usr/bin and /usr/local/bin. It should be clear:
- /applications – here we’ll install user space applications
- /configuration – here we’ll save all the user space configuration files
- /documents – here we’ll save all the user’s documents
- /system – this is the system directory – this directory is not accessible by anyone else than drivers and kernel
- /system/configuration – here we’ll save all the system configuration
- /system/devices – this is the directory where you can get access to some devices
- etc.
- No POSIX – as I have some ideas of my own on how to create APIs. POSIX could be an abstraction layer over my APIs for sake of compliance. For example no stdlib or stdio by default. Instead of malloc() – i’d prefer something called void *memory_allocate(size_t size, permission_t perm) with some specific permissions (like weather this memory block can be shared across threads or processes) and instead of fopen() i’d prefer handle_t *file_open(char *uri, mode_t mode) where mode is not a character, but some kind of a structure. It’s more about the taste of the API naming;
- Last but not least – GUI from the start. Tell BIOS to display a cute logo and try to avoid any flickering – so that logo is faded out only when kernel has readied it’s video processing drivers. (I know that you can read from video memory as well as you can write). Yeah – Apple style!
My research
What I’ve found out so far:
- Boot sequence:
- BIOS;
- Master Boot Record (MBR);
- Bootloader – the program that takes it over from MBR and loads your Kernel;
- Kernel.
- How to write your own MBR and write it to Disk on windows. I’ve written a small utility in Visual C++ that allows directly to read/write from disk (download here, source included for Visual Studio 2010 Express);
- How to write bare bones C kernel entry point.
- How to write “naked” functions on Windows in Visual Studio
Missing link – I still don’t know how to properly step from MBR to Bootloader to Kerlen, that is – write your own MBR code that would load bootloader, pass the execution to bootloader that would load and pass execution to bare bones C kernel:
- What exactly is Global Descriptor Table (GDT) and Interrupt Descriptor Table (IDT), and how it looks in C and Assembly?
- How and when, and again how, if when is later (for example in Long Mode or Protected Mode) to set up all this GDT and IDT stuff. They say you have to set it up before the kernel. Then they say you can set it up with dummy values and set it up later in kernel. Then they say, that to set it up, you have to be in Real Mode, so your kernel (which might be way over 1Mb of real mode space), needs to switch between modes. And then if your kernel is over 1Mb, you can’t access memory locations after 1Mb, and so on… It’s confusing, but I’m going to find it out and post it here later on.
- How to handle Interrupts in C? Will they perform as callbacks that await some return values or do I have to use inline assembly to process them correctly;
- Is it possible to write MBR in C? I do understand that you still have to set ORG to 7c00h, and use some specific assembly instructions, but if they could be wrapped in C using inline assembly and C entry point can be glued with few lines of assembly code, why not?
Usefull tools and must read
Here is the list of tools I’m going to use:
- Visual C++ 2010 Express Edition – I’m using Windows XP, so there’s no VS2012 Express for me 🙁
- Netwide Assembler
- Bochs – x86 emulator, so that you don’t have to write to disk, connect it , boot up, see error, shut down, rewrite your kernel, write to disk … (you got it)
- Cygwin – just in case you need some great Unix tools on Windows
Must read:
- Definitely OSDev Wiki – it has some great tutorials and hacks, although the attitude all over there is “wannabe rough-and-tough”. Then again these people there are coping with some kind of madness after what they’ve gone through 🙂
- I’m currently reading this tutorial from JamesM – it’s marked as “Hard. Good luck”, but it explained some things to me quite well.
Huston we have a problem.
On the side note – It’s 21st century, but our PCs are still booting up as old-fart Intel 8086. It’s been 33 years, but our computers are still booting up like it’s 1979. Why the heck Intel, why? Actually I have a suggestion for transition phase – just add a jumper to the motherboard and create some kind of Super Mode (just like Real Mode, but with full 64-bit memory and instruction support). When the jumper is connected we have the old Real Mode, but once all the OSes are ready to do some modern booting, disconnect the jumper and run Super Mode instead. This way, modern OS boot-loader can easily work with all the resources on the computer, and not some A20 gates and jumps from Real Mode to Protected Mode. And then there’s BIOS and 512 byte MBR. I know that 512 bytes is enough to look up a partition on the disk and load some boot-loader code. And then there’s x86 memory layout with some holes, so you can’t just read up some 900Kb from disk directly into memory without crashing.
Anyhow that’s the world we’re living in, we have to cope with it and we have to do a lot of tricks to create our OS as modern as possible.
Hi,
I think it is a great initiative 🙂
I also hope to read about your future successes or failures in the future in this blog 😉
Good luck
Definitely 😉
Take a look at https://github.com/captain-protect/mysmall-OS . It’s made just for fun, but may be you will have some starting point.
http://www.returninfinity.com/pure64.html
I would take a look at the Pure64 project which fits the bill for loading the system into a barebones state. Open Source to boot.
Whoa! Thank you Denis, somehow this one didn’t cross my eyes. I’m already working on a port of one other solution (found on OSDev, I’m aiming for VS2010 Express with Cygwin) of which I’ll post the results here. Then I could try this one out too. Also if I’m not mistaken Pure64 is BSD licensed, right? The license is one of my concerns.
It is 3-Clause BSD license essentially but doesn’t have BSD explicitly written into the licensing.
What about SharpOS?
Sorry, but C# is not what I’m aiming for right now. Of course there are many different approaches on OS development (Java or pure Assembly, for example), but I’m aiming for classic, raw power yet readable code.
Hey gusc,
This is wonderful. Any reason why you are choosing to build this on a windows vs a mac/linux platform?
-BJ
First of all I’m used to Windows, although my daily work is targeting Linux servers (web development), but the development still resides in Windows.
Second, there are plenty of tutorials for linux users, but there are almost none for Windows. I just thought it could be useful to show other options (than Linux) to fellow developers.
So I just thought that I could accept this challenge and share the results. I still don’t know what will the outcome look like – it’s work in progress with a lot of unknowns. For example I just found out that VC compiler does not support any other format than PE. So my next post will be about setting up VS2010 to work with Cygwin and GCC.
Why a C++ compiler? It only works as a C compiler inasmuch as C is mostly a subset of C++. There are plenty of good C compilers out there, and some of them even integrate with VS
Not the compiler, it’s the IDE thing here, and as I wrote in reply to the previous comment – i’m working on Cygwin + VS2010 solution, so that we have GCC in Visual Studio Express
Best of luck to you! I’ve been delving into this over the last few weeks myself (just as a hobby), and here’s some advice:
* Read Intel’s systems programming manual, and familiarize yourself with the x86 instruction set. You said you don’t care for assembly, but you will need it every so often.
* I would start off using an existing bootloader; it makes the early kernel development stuff a *lot* simpler! After a while, you can write your own.
* Don’t worry about the GUI stuff (at least not yet!)
Thanks!
And to answer your advices:
* Check! – although the full manual is around 3k pages, I’m currently reading it as a bedtime literature 🙂
* Check! – see my new post – I ported Bare bones bootloader from OSDev Wiki to Visual Studio and posted the solution file with some extra build scripts and a step-by-step manual to set everything up;
* Check! – currently I’m more into the excitement of doing something new, although GUI is one of the major things I’ve commercial interest in.
Very cool. Why are you doing this? What is the motivation?
Mostly curiosity and a regular “challenge accepted” for everyday developer. But if it plays out well, I could use it for some of my commercial ideas.
Can you share those commercial ideas of yours? 🙂
First of all, you’re better off writing one for the architecture of the next decade – ARM (ARMv8 specifically). Intel is already on a decline path, which will accelerate in the coming year. You don’t have to believe me. Just start paying attention to their next quarter results.
Second, get ready to pay some licenses to Microsoft as soon as you intend to make it commercial. And no, the fact that you’re not trying to copy Windows at all is not relevant. They WILL come after you. So keep that in mind.
+1 on both points. I would target ARM/mobile devices.
Good luck anyway !
I’m looking into ARM too, but for now it does not deliver the power I’m interested in. For now I’m looking for all the power I get from the CPU as I’m targeting realtime data processing. As for Microsoft, well, I would even call my idea an OS, as it’s targeted into very specific field of data processing and not for general public (a.k.a. Personal computing).
Definitely go with ARM. ARM64 is on the horizon, and it’s very easy to roll your own highly paralellized ARM systems. AMD, for example is going to start offering 64-bit ARM chips. x86 is dying… Look at how the architecture struggles to scale to 4+GHz and 8-fold parallel chips.
In short – parallel, ARM, is the way of the future.
All of your bootloader nightmares will go away with ARM.
All current CPU’s have difficulty reaching those clock rates. In fact, ARM processors are nowhere near that clock rate. (Mostly due to power constraints.) Not easily surpassing 4GHz is not an x86 issue at all.
Furthermore, x86 doesn’t seem to have issues scaling to multiple cores. Take a look at Intel’s Xeon Phi CPU’s which currently ship with 60 cores (IIRC).
I suspect x86 isn’t going away anytime soon. If anything, ARM cores are getting more similar to x86 cores. (For example adding OOE.)
The Xeon Phi is a failure that was renamed three times before being offered as an 8 times slower alternative to GPGPU.
I really don’t think that’s a testament to the awesome scaling of x86.
Other than that, x86 doesn’t have scaling issues more than ARM does, and in fact probably less because the ARM requires a more complex interconnect due to a much higher core count.
Either way the reason ARM > all is the licensing model that enables anyone and their dog to start selling CPU’s tomorrow, thereby creating a healthier market for the consumers.
LOL.
I really get a kick out of the Intel bias. There is nothing really hard or special about writing boot code for Intel. Or any code for that matter.
x86 struggles with parallel? What about this:
http://en.wikipedia.org/wiki/Titan_%28supercomputer%29
I guess until you get over 18,688 cores it doesn’t count as parallel?
I think this is a great project. You will learn a lot from it.
Waye
Have you heard of Baremetal OS? It seems to share your goal of just running code as soon as possible. The source code is on github (but it’s written in Assembly): https://github.com/ReturnInfinity/BareMetal-OS
Yes, I heard about it today – see the comment from Denis about Pure64.
“modern” and “USB” in the same sentence? you mean thunderbolt and firewire, right?
usb is good for keyboards, mouses and small flash drives. anything else needs a decent connection.
You’re a Mac user, right? 🙂 I wouldn’t put “modern” and Firewire in one sentence either. Firewire is just 15% faster, and it’s used mostly for audio/video transfer (although there are a few hard drives on the market too), on the other hand USB is “Universal” and the market is flooded with USB devices. Thunderbolt, yes it is modern, but it’s still a technology of tomorrow. I haven’t seen any PC boards comnig with TB ports yet, and there are no devices that utilize this technology yet.
Furthermore, Thunderbolt is bullshit, external PCIe is the future and it has no shiny buzzword or intel tag.
(which thunderbolt is an implementation of more or less)
apropos directory tree
Probably a bit early, but since you have the chance:
Why not rid the OS entirely out of the way, into one hidden directory:
/.os/programs/
/.os/configuration/
/.os/…
And let ‘/’ be the user’s directory?
Then, the user can create directories like /poems/ and /applications/ (for job applications) if that’s desired.
Think of a multi-user OS… you already got your answer.
I thought of a multi-user OS, and I find your response to be lacking.
If User1 logs on, he should see / as his storage area.
If User2 logs on, then he should see his own / as his storage area.
There is no reason to actually use / the physical version. The OS should just use / the logical version.
You have no idea what you’re doing and you’re going to fail miserably. GUI from the start (nice BIOS logo, seriously?)? Directory structure? You’re focusing on unimportant details, you’re a kid who wants a fancy toy (nothing wrong with that — but you have to know what you’re doing first!).
And your accomplishments are? Don’t drag other people down. It’s his project and as he said before, he has a lot of unknowns, he was just listing the stuff he envisions in his future project.
He, who makes no mistakes, makes nothing…
Seems to me that’s how you find out what you’re doing – by experimenting. I think this commenter has things backwards – don’t let his douche-laden negativity hold you back.
How do you think all of the other great developers got started? I’m sure you believe they just already magically knew all of it.
Those are not ‘unimportant details’, those are ‘important fundamental design features’. The extreme impossibility of having those features in mainstream OSs demonstrates the fact that they are built on 30-year-old craptech.
It is a visionary that is willing to say “all this is bullshit, I’m going to do something fundamentally better”.
Go for it, my brother Gusts.
1. What exactly is Global Descriptor Table (GDT) and Interrupt Descriptor Table (IDT), and how it looks in C and Assembly?
Both the GDT and IDTs are tables: Chunks of memory bits put in a specific layout. Think “lookup tables”. You tell the CPU where in memory these lookup tables are, and the CPU goes ahead and looks things up in them. In C you can think of it as an array of structs (of a specific layout). For x86-64, take a look at the Intel manual on the subject, it is fairly detailed.
2. How and when, and again how, if when is later (for example in Long Mode or Protected Mode) to set up all this GDT and IDT stuff.
Note: I have only written code that jumps *directly* into long mode without going through protected mode so the below may not apply to you.
IDT: The first thing you do is disable interrupts (cli). Once you’re in long mode, then you can go ahead and set up your IDT properly at your lesure before reenabling interrupts (preferably very soon after entering long mode).
GDT: You only need a long mode GDT. You will want to jump to the correct code segment *immediately* after entering long mode.
PML4: You need one in order to get into Long Mode, so have it ready before entering long mode.
What I would do is:
1. Enable A20 line.
2. Sanity Check: Detect if CPU supports long mode (otherwise crash).
3. Setup the stack.
4. Detect memory.
5. Prepare PML4 and other paging related structures. (At minimum you’d want to identity-map the portion of memory where your code is sitting.)
6. Disable all IRQs. (out 0xFF to ports 0xA1 and 0x21)
7. Load a zero-length IDT (just load in a IDT descriptor with address 0 and length 0. Why? Because if you receive a NMI during long mode activation, you’d want the computer to triple fault.)
8. Setup paging (Set PAE and PGE in CR4, Set CR3 to PML4.)
9. Enter Long Mode (First set LME then activate PG and PE in CR0.)
10. Immediately load GDT and jump to 64-bit code.
11. Well done, you’re now in long mode.
12. Set up IDT and APIC (IRQs) and enable interrupts.
3. How to handle Interrupts in C? Will they perform as callbacks that await some return values or do I have to use inline assembly to process them correctly;
Write some assembly to setup the stack then call into C code. Doing so is trivial and it’s only a few lines. Interrupt handlers must be performant and return as soon as possible if not immediately.
4. Is it possible to write MBR in C?
Yes and no. Space is the major issue, so you would probably want to handwrite it in assembly.
Even if you plan on making your own boot loader, if you start out just making a multiboot kernel, you can use the -kernel feature in QEMU to load it directly.
You can also launch a multiboot kernel directly from grub. The entry point can be written assuming long mode has already been setup which lets you start out with very little assembly code. It’s a nice way to boot strap yourself and ease into the lower level assembly bits.
See http://www.gnu.org/software/grub/manual/multiboot/multiboot.html
I’ve been wanting to get into low level development for quite a while! So far, I only have experience in C++, Python stuff. Recently started re-learning C. Would love to get involved in any way possible! 😀
I just finished the Baking Pi tutorial. It is for the Raspberry Pi and it is in assembly but you actually create a really basic operating system. I learned so much about operating systems from it and I feel it would help you on your journey.
http://www.cl.cam.ac.uk/freshers/raspberrypi/tutorials/os/
You lost me when you said “Visual C++”
You don’t like it?
Consider looking at 9Front: http://code.google.com/p/plan9front/
9Front’s goals are somewhat congruent with your plan. At the least, you may find the kernel source concise, very readable and clever.
Best regards,
Vester
The first thing that sprang to my mind when you said “C++ for GUI” was Qt – is this the plan?
The comments in http://code.google.com/p/nix-os/source/browse/sys/src/nix/k10/l64sipi.s?r=6ba826535859dc3534915066040016e79decd8ce
(“Welcome to 1978”, “Welcome to 1982”, …) but the code shows a fairly short path from 8086 to amd64 (x86/64 to you), once you decode the idiosyncratic assembler.
Thanks for the inspiration. BEFORE you DECLARE the ‘design basis
philosophy’:
1.)where is the REAL NEED of Open Source?
IMHO, it is RTOS for FreeBSD. small, embedded devices.
2.)do NOT re-invent the wheel with POSIX, etc. Study BSD and Linux
evolution.
u can easily reach ycombinator ‘hacker news.’ I use elinks, egrep and somtimes
I CAN UNDERSTAND THE conversation thread. sometimes not ( i am not just
smart enough).
3.)WHEN IN DOUBT, go ML, OCAML, then Haskell, then Ruby, Python,
C++ (with tested libs), C, the assembly. OR
IMHO, you seem to be going C only. Great!, use C ONLY, not including Python.
4.)much better to do a very small OS with limited functions and add on
features, then a BIG COMMERCIAL bells and whistles and fancy UI and KDE
and all sorts of bloat.
5.)network: including SCTP and TCP and others?? how about ip version 6,
and 4 and others?
6.)IF OS was so easy, then FreeBSD and other BSD would not be labeled
an ‘intermediate’ or advanced OS. IMHO safer, more stable, but more
advanced than Gentoo Linux… not trying to start flame war IMHO.
7.)obviously, ZFS rocks! but crypto, tunable, extensions… etc seem
to me (again I am not a GENIUS in this area) to be ‘black arts’. Modular works,
passes basic tests, but SURPRISES and bugs and ‘threads’ may be lurking
as MONSTERS in the dark.
If you want to avoid writing your own bootloader and assembly glue, consider using Linux to kexec to jump into your own C kernel.
https://en.wikipedia.org/wiki/Kexec
(At some point hopefully Coreboot, GRUB, and the Linux Kernel will realize they are the same project.)
for the missing links:
1. You can make C structs for GDT and IDT by bitfields like this: https://github.com/Fleurer/fleurix/blob/master/src/inc/gdt.h
2. GDT must be set before you entering the Protected Mode, you can set up a GDT in your bootloader temporarily, and set another GDT permanently it in C.
3. follow jamesM and he will tell you 🙂
4. Yes, xv6 use C to read sector in their MBR. But a small piece of ASM code still can not be avoided, before entering the 32bit-Protected Mode you have to start up from the 16bit-stone age, for asm it’s easy to mix up 16bit code and 32bit code, but it’s impossible or hard for C.
not impossible and not that hard.
you can use inline asm code in c,
or simply call a function pointer to a string, containing the pre-assembled binary.
example:
main() {
int vendor_str[4];
vendor_str[0]=(*(int(*)())(int*)”\x31\xc0\x0f\xa2\x89\xd8\xc3″)(); //xor eax,eax;cpuid;mov eax,ebx;ret
vendor_str[1]=(*(int(*)())(int*)”\x31\xc0\x0f\xa2\x89\xd0\xc3″)(); //xor eax,eax;cpuid;mov eax,edx;ret
vendor_str[2]=(*(int(*)())(int*)”\x31\xc0\x0f\xa2\x89\xc8\xc3″)(); //xor eax,eax;cpuid;mov eax,ecx;ret
vendor_str[3]=0;
printf(“vendor: %s\n”,vendor_str);
}
Whoa, this is neat. So it’s basically inline binary (not even inline asm)? Would this work if rodata is stored in execute disable bit area (my guess – not)?
When people says NO, YOU ARE CRAZY, ETC, don’t listen them and keep going forward.
Best of lucks on this project!
Cheers
Hello sir
i am ready to work with you << Because i am too enthusiasd in how OS are working <<< I know Python,C,C++,Java, Little Assembly << my projects are given below.
1 http://code.google.com/p/linux-intelligent-ocr-solution/
2 http://sourceforge.net/projects/sbrw/
3 http://code.google.com/p/angela-open-talking-typer/
Thanks
Why not use javascript running on V8 for your scripting language?
I totally agree. V8 is fast enough and JavaScript is very, very popular with a lot of mature libraries. You could even copy the node.js API (oh what a dream it would be).
You should really spend some time (weeks at least) exploring the wide world of OSes. There have been many and the ones you seem to know are only the most conventional and most design-compromised. There’s a wikipedia list of operating systems (http://en.wikipedia.org/wiki/List_of_operating_systems) which includes a section of “hobbyist developed” ones!
If you want absolute simplicity you should read about Forth, which comprises an OS, programming language, and libraries in no more than a megabyte of source: http://en.wikipedia.org/wiki/Forth_(programming_language)
Plan 9 was written by some of the people who created UNIX and wanted to get it right this time (http://en.wikipedia.org/wiki/Plan_9_from_Bell_Labs).
You focus on directory structures in UNIX-like file systems. There have been file systems organized in quite different ways with different central ideas. Besides Forth’s non-file-system file system, there was the Be OS’s BeFS which associated unlimited key:value pairs with every file. This eliminates most need for a directory structure at all, replacing it with the ability to quickly select a list of files matching a key. Like all the kernel private files would be tagged access:Kernel, or you could instantly list all the files language:C project:newOS instead of doing cd /my/newOs/src .
So, good luck, but expand your horizons before you really dig in!
Great initiative.
I wish you best of luck with this. It is something that I too would like to one day attempt to create, if for no other reason than to have fun while learning a whole lot of things.
I really like some of your ideas and I will definitely follow your blog to see how you progress with that.
Only advice I can give is, first take patent for this.
oh god! patent …. huh. … you should be joking
Lol, patents. Why should he do that?
Even without licensing he’s protected by copyright law.
And I honestly doubt some huge corporation would want to steal his idea.
Best of luck with your learning journey. As part of a development reboot for Belenix, I’m working on setting up some good release engineering, and will be looking at your project to cross-validate my ideas.
Please also keep careful notes of your lessons learned on a daily basis. These notes could become the foundations of a book someday.
I like your idea of wanting to get into C code as early as possible. This will help with ports to other architectures too.
EPIC FAIL! :ROFL:
If you don’t understand the difference between /bin, /usr/bin, and /usr/local/bin, I wish you luck. You’re going to need it.
Yeah, I thought the same thing.
Although, he seems to be a Windows user, not really experienced with Linux (Visual C++, etc).
Nice one! 😀 Good luck with your endeavors.
* useful
* Houston we’ve had a problem
Fixed it for ya.
Your project sounds great. Good luck!
I would suggest Javascript as scripting language for user applications. It has huge traction, there are several engine choices already available, it fits perfectly with your URL paths and besides, you are going to need a javascript engine for an eventual browser anyway.
My C days are way behind me, but I feel kind of envious: every computer programmer has dreamed about building his own OS and you are actually doing it.
Best regards from Spain
Please learn about the filesystem before denegrading it. I would recommend http://www.debian.org/doc/manuals/maint-guide/ and http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html
The key points that you are missing are: /home/$username are essential to give each user a full environment for themselves, so that multiple users can securely use the system without interfering with each other. A user has no business working in filesystem root – he has his own $HOME. /usr has the mayority of the software and software data, BUT it is assumed that /usr is large and might be unavailable during system startup (for example, if it is mounted over a network).
Forget about GUI. Make it work first, *then* add bells and whistles.
Obviously you have come from the broken world of Windows programmers that barely have an idea what is and is not an operating system. Learn a real operating system first – build a Linux system as your primary desktop from scratch before your try and improve it. Unless you understand what has been the state of the art for the past 30 years in operating system design you are just bound to repeat all the old mistakes.
Also – forget all about any tool by Microsoft. They are useless compared to the real stuff. You will have much more trouble trying to force it to work right. Learn to use gcc, as, make.
I would recommend starting with the same target that Linus Tornvalds first set himself: boot, start two processes – one that prints ‘A’ to console and another that prints ‘B’ to console.
Ha-ha! “How rude“! 😀
Good, now I don’t need to comment.
Thanks.
Lastly, forget about Cpp, there’s a reason Torvalds dissed it, and there’s a reason Carmack acknowledges being “forced” into using it for id against his better judgement.
Every language sucks, but at least, C is fast at sucking and gives you maximum control, a requirement for an OS.
Your directory structure doesn’t seem to be well-suited to multiple users.
Agreed.
The directory structure seems single user minded which basically takes the modern design into early Windows-esque design to not support multiple users from the beginning.
A Mac OS X style perhaps: /Users/*
Go for it. Maybe you are the next Linus Torvalds.
That’s insulting to Torvalds tbh. He could see the difference between a broken tool and a working one.
…and his hair color is different! Apart from your judgement being totally subjective, even if it were true, why would it have mattered so much? Should someone have exactly the same skill-set as Linus Torvalds, to be “the next Linus Torvalds”? I think even talking about the possibility of being like him is a compliment to him on its own.
Your post seems to have inspired a lot of discussion over on HN.
http://news.ycombinator.com/item?id=4833218
I won’t say good luck. Just do it, mate.
Hi,
First of all, let me start my saying, this is a great exercise and I wish every engineer/developer/ take on such endeavors.
Secondly, and not to sound rude in any way, May I ask why? You seem to have some interesting ideas but as to every solution there is a problem that it intends to solve. What is the problem that you want to tackle?
You definitely point out that the mainstream OSes out there have a lot of legacy code. Which is not ideal and I would whole heartedly agree that it should be ripped out and replaced with better and cleaner pieces. So, IMO the problem is to refactor things to push out the old and put in the new.
Also, an exercise that has helped me a lot in the past is called story mapping. I don’t want to impose any methodology but rather just want to give my two cents.
Also, I would love to know the answer to my question above.
I mean to write my own OS. some day. or at least be the lead on that OS dev.
But quite frankly, I believe the first step is compiling the tiniest linux you can make, and then make the source better, and then start wondering if you should rewrite that.
Writing an OS from scratch without having any clue what an OS is, is a bad idea.
Also you should learn BIOS editing before you even start tinkering with boot loaders. I think you may stumble upon even better news than BIOS>MBR>OS 😉
I have the same idea of building my own OS a decade ago but didn’t go into that, I glad to see you have the passion but I’m aware it will remain a hobby OS instead of commercial because MS and Linux has the largest community on both free and not free.
If you were to be different from them, I would suggest you to develop an “OS Framework”, encourage developers to customize different type of OS such as Kiosk, Gaming, Desktop, Thin-Client, etc by drag-and-drop services and components.
The priority is to get Google Chrome, Firefox, Flash and Java working on your OS is that matter.
Lastly, I wish you could add a good support for MIDI port as well, it is ideal if you could get MIDI data to sent to network server thru message passing, this make life easier for developer. If you interested how MIDI work, “Java MIDI Resource” has a comprehensive tutorials for that.
Have a look on the excellent boot loader XOSL (gui / mouse / keyb / big screens support from boot) which could be a serious basis for hacking an OS from scratch
And on the complete OS KolibriOS as well … GUI, network, unicode, etc. 2.8MB only …
I think it is insane to write an OS in anything but a sort of assembler language.
C and C++ are NOT assembler; some uninformed guys would like me to believe that they are the same!
If you’re writing some kind of kernel-based OS with other utilities and so forth bundled around it, you can get away with writing the primitives in assembly language, and the higher-level functionality in C… but you’re right: mainly, assembly language is the way to go… especially if you want something which is properly fast. Those couple of clock cycles you save in a few of those often-called, low-level bits of the operating system (request memory allocation, for example) are going to make huge differences once their usage begins to scale up – especially in a multitasking environment.
Hi! I’m junior C/C++ developer with 1.5 years of commercial exp. Is there some tasks that will help your project that junior can do?
First of all best of luck. This will be a great learning experience, and I look forward to your updates.
Have you considered which filesystem(s) you will support? Not the directory structure which has been discussed above, but the data structures stored on the disk. It would be quite a bit of development to support NTFS without building on an existing open source project (not to mention it is proprietary to Microsoft). Maybe FAT would be an easier filesystem to target at first?
Yes, for now i’m treating the hard drive as one big chunk of byte array. In the future i’ve my eyes on extfs, but that depends on licenses and possibilities. I need a FS that also incorporates version control – so once you write to file, the old version (or more precisely it’s diff) gets stored in a version tree.
Good luck man!
It’s sounds completely unreal, but if you’ll do the basic and the core API lightweight, clear and fast – I believe it’ll have a future!
Good luck man!
You said that this system will be “targeted into very specific field of data processing and not for general public”. What does that mean, exactly? What will be the purpose of this system? Also, I like your idea of directory tree (I don’t see the difference between /usr/bin and /usr/local/bin too 🙂 ) but it seems to be a bit single-user oriented. Will there also be something like “home”directory for each user?
There were some suggestions for multi-user system in comments above, but for now i’m targeting single user mode, thus – only one “home” directory (a.k.a. documents). If you look around – how many computers are being used by more than one user? Yes of course, you need atleast 2 – one with super admin right, the other one for the regular user, but, for example, iOS and Android systems are multi-user (I think), but with only one user up-front. And mostly it’s relevant for corporate use – which is not my case.
As for “data processing and not for general public” – it means, that not a lot of people will use this system, because it’s only for specific needs.
So it will be single-user system with support of a narrow range of devices but designed to be as fast as possible. It seems that you have some specific task in mind for that system, so i wondered what is this task?
Sorry, but I’m not willing to disclose that yet. Maybe when I’m ready for alpha/beta testing.
привет, ты идиот. но это нормально, т.к ты из Латвии.
Привет, ты идиот и это ненормально. Сходи полечись.
хехе, кого ебёт чужое горе 🙂
I’ll risk being pointless by adding yet another item to the “hey you should totally look at this”-list.
As your focus is on modern tools to build systems, I would advise keeping Mozilla’s new programming language “Rust” in your radar (rust-lang.org). It is built by the very systems people who are using C every day, both languages interface nicely, but Rust takes a different approach to systems which I think will work better for modern systems. It is super safe (you can’t even cause segfaults on purpose, a little too safe for my taste), uses a clean syntax, and has good multi-threading. The only problem might be that it’s still in early development.
I forgot to mention that it can also be used as a scripting language, since the rustc compiler is able to do JIT and can act as an interpreter.
Many people have taken a stab at this. I hope you both succeed and enjoy it. Here’s some minimalist OS’s for you in case they’d be useful or help you figure out how to solve problems. I’ve used one or more in the past to remove bloat (read: Windows/Linux) from small projects.
RTEMS
http://en.wikipedia.org/wiki/RTEMS
Classic single process OS. Small. POSIX compliant where possible. (Hey, you can always change that… 😉
http://www.dex-os.com/
Game console version of your project. I once used it to jump straight to my code. Oh did my system run better.
http://www.menuetos.net/
They wrote most of this in assembler. Tons of functionality in tiny amount of space. Notably, they got plenty of graphics you might borrow.
http://ecos.sourceware.org/about.html
Maybe shred this thing to bare minimum, then work from there upward?
http://www.minigui.org/en/
That and directfb can also help for graphics support/inspiration.
Hope some of this helps you!
Have you thought about:
-task scheduler
-driver model (micro or monolythic kernel)
– filesystem
– memory access and management
– IPC
– threading
– users…
I think that this is the kind of task that doesn’t begin coding. With difficult and low/level implementations like that you want to minimize design errors before coding.
I don t know your background, but you can take a look at simpler embedded device operative systems like TinyOS, Contiki and why not, an early Linux, either getting involved in kernel developing or at least reading the Linux Kerne Development book by Robert Love.
Good luck!
Getting one of the earliest simplest linux sources could help too.
Are you planning to build an exokernel ?
http://code.google.com/p/xomb/
I did this, too — a very long time ago — on an old 386 machine, all in assembly language. The idea was to try to create a multitasking DOS. It worked to some extent; more importantly, the exercise of doing this had inestimable value: I can’t over-emphasize how important it is for a programmer to do this at some point, just to do it. Good luck with this: I wish you all the best.
I higly recommend to have a look at the cosmos project (http://cosmos.codeplex.com/)
If you’re going to be writing an operating system from scratch, you might want to look into using something better than C. It might be something that doesn’t exist yet—the way I understand it, C was made to write Unix. C isn’t all bad, but using the preprocessor for importing libraries and some other things aren’t so great. It might be kind of cool to use something that has Common Lisp- or Scheme-style macros at compile time while still giving you low-enough-level access to the machine. Good luck!
For a scripting language, nothing will beat Lua + LuaJIT. I highly recommend it. It’s a beautiful language. And the stock implementation is beautiful as well, if slower than LuaJIT.
And good luck. You have a long, hard road ahead of you. My only suggestion, that I think will help the most: stop being afraid of assembly language. Learn x86 assembly, and learn it thoroughly. Even if you don’t use it much, it is incredibly important and a must-know thing at the bottom levels. Through learning x86 asm you’ll also learn a lot about the architecture, especially as you’re using the Intel manuals.
You’re too dumb to realise this now. But after you build your crazy OS. You’ll realise how much time you have wasted! All I the things you talk about here have been thought of already. Your OS isn’t going to be good enough for anything but show off! And the only ones gullible enough to believe you would be other dumbfux
Someone already mentioned Xv6 – be sure to check out its sources. Also, this course might be helpful: http://pdos.csail.mit.edu/6.828/2012/
Have you thought about using something like VMware Workstation. Basically this will allow you to run your OS on an x86 virtual machine while still preserving your development environment.
Awesome project idea 🙂 Ditching legacy definitely sounds like a winning approach given your goals.
One suggestion: take a look at Lisp Machines for inspiration; there are some good videos up on YouTube (e.g. http://www.youtube.com/watch?v=9whxPd4haKc).
The particularly awesome thing about Lisp Machines is that they were Lisp all the way down – which means that you can inspect, change and debug any aspect of the machine’s behaviour in an interactive Lisp environment.
Imagine being able to do that on a modern machine. Word processor crashed? Oh well, no worries – take a look at why, fix it (perhaps by redefining the offending function, or whatever) then carry on. Want to change the way window switching works? Ditto – inspect it, fix it, carry on.
(Although it wouldn’t be possible to do ‘all the way down’ on an x86 architecture it’d at least be possible to do ‘most of the way down’.)
Gusts, what license you want to use? GPL, BSD or your own license?
Thanks.
Probably some parts in BSD – I like BSD, because it allows you to use closed source together with open one in a single binary.
> some parts in BSD
> some parts
Entire Your OS will be proprietary, isn’t it?
For now I’m going to post stuff I put together on this blog. The final product is a long way ahead – so I don’t want to rush my decisions. Maybe the kernel will be BSD, but userspace – proprietary. Maybe I’ll have a monolithic kernel and some parts of it will be BSD, some proprietary. Maybe I’ll just write a framework to build simple applications for single-application environment… I still don’t know. I’m researching and learning. 😉
it’s great news.
Good luck!
Good luck! 😉
Every one is excited about your idea. But I am going to give you some valueable opinion.
1. You are telling that you are going to ditch legacy standards. Thats a good point (for experimentation). Legacy is the one thing that keeps Microsoft in the Market.
2. I do not think Windows is going to last any loner than 10 years from now. Windows is predominant in desktop PC’s only. but recent actions of Microsoft reveals that, they are moving towards mobile computing. Another one thing that Win8 ridiculously reduced support for games. I assumed that you are trying to help Windows people.
3. Writing you own Kernel. Hats off. that is huge undertaking and you are so confident. I appreciate it.
4. Linux is not legacy kernel. One of the awesome things about Linux is that, it is modern in all the way. Supporting from Calculators to Super Computers. You eliminate what ever the code piece you want from the kernel while compiling. thats most embedded system vendors immediately choose linux as their kernel.
5. GUI from the start. I am assuming that GUI inside the Kernel like Windows. Honestly not a good idea. there is no resultant advantage in porting GUI code into kernel. I think you are an engineer. You know the trade-offs
I plannned to write my own OS custimlt built for programming environment. but not from the scratch. use whatever is available. I will tailor them to my needs. I strongly believe in one thing. Criticism which is positive, never been useful.
5. But why there’s still no kernel drivers for video in Linux? Why is everything run through X11? I know there’s framebuffer driver, but as I understand it doesn’t use GPU accelerations – it’s just plainly swapping buffers and all the processing is done by CPU. Am I right?
Already tried that. Why don’t you contribute in existing kernels? Like Linux or BSD? It will be more useful to you and to everyone.
If you are making your own OS, then you have to make it better than both Windows and Linux, if you don’t want to ruin your efforts. Plus you have to do something new, as there is no point in re-inventing wheel
Imagine giving that advice to Linus back in the day …
“You know, that Linux thing is a total waste of time, re-inventing the wheel and all. Why not just contribute to MINIX?”
Actually people wanted to contribute to MINIX to make it big like Linux is, but Tanenbaum refused changes that added advanced functionallity (for the time) so that a student could understand all of it in a semester.
Sounds like superb stuff… Start it… let us contribute to it… let’s see what we get…
I even liked the WebOS… built from scratch… entirely written in JS.
Good fortune to you. I followed the same path by necessity. I work in embedded systems and there have been many times when my employer or customer was unwilling to pay for an OS. I really didn’t want to write single-threaded code, so I would write my own real-time kernel with some minimal OS features. It was a great way to learn.
You can build real products in the embedded world this way, maybe you can consider that direction.
Best of luck,
-G
Hey,
Its a great initiative and surely would be a helluva ride 🙂
Enjoy and I will be visiting here very often.
Also, I would suggest, why not try to include this open source project as a Google Summer of Code, 2013? It will certainly allow many others to learn, contribute and experiment!
All the best
~Vikash
Did you have a look at http://www.reactos.org?
Yes a long time ago, but I’m not interested in Windows clone. My OS will never have a corporate use, neither it will have an universal use (like a desktop OS)
Wow, realy great ideas and I agree.
Just wanted to note – it would be really cool, if your GUI would be HTML/CSS based, and backed with mentioned Python – app development would be such a breeze! I haven’t tested it yet, but Win8 uses HTML for GUI – that would mean, that maybe your OS could launch even Win8 software!
And last, I have always dreamed of some Super-OS, that could launch software made for any other OS out there. It would be cool if you would give support for Windows and Linux software to be launched in your OS. That is big issue – if your OS won’t have software, noone will use it, except some dedicated people out there.
Good luck – will try to follow your development and spread the word.
I’ll probably need some web browser anyway, because that “fraction of general public” will likely have a need to browse some things up on the net while working with my OS. So my eyes are on Chromium, and while we’re at it, it could bring node.js too for JS as a scriping language … but I’m not a big fan of JS – it’s OK (not great) where it is – on the web, but I still don’t feel the need for it anywere else. I also read somewhere that node.js has performance issues with C bindings, that’s why database drivers are written in JS.
Did you look at Tennenbaum;s books? Also MINIX might give you some ideas. Intel has some good documentation on their web same as IBM but you probably already saw these
Good start…. make sure you file a patent 🙂
Good luck! I want to write the programm for you OS 🙂
great thinking.
I also want to be a part of your project. If you think I should also contribute in your project work , so I am always ready for that . It’s my honour to work with you.
so if there any task that you want to accomplish from me. plz contact me on my email-id “abhilash.shingane5@gmail.com” and my contact no is 918446283182 .
> make my own and allow only my own OS on the system
You just eliminated most of your prospective initial user base for no technical reason. Good job.
It’s a decision for my prototype, so that I could learn all the ins and outs of boot process + I wouldn’t have to deal with some 3rd party bootloader APIs and rules. I want to find out what can be done from the very first instruction that’s read from my disk.
I hope you don’t think to make it all alone? Love your idea, but I suggest you make some working prototype ASAP and attract more people to your project.
I also hope that you intend to make OS for users, not just some kind of research project, so you’ll be thinking through graphics stack, performance, ease of programming for your OS, etc, etc.
Good open-source (you are opening its code, right?) desktop system is yet to be created, IMO. And while there are talks of desktop dying out, it’s not going away anytime soon.
Finally, why do you want to disallow any other OS in user’s system? Doesn’t make any sense.
Good luck, hope to see some updates.
Have you scoped this project out? Benchmark:
3 smart CS seniors — 1 operating system:
Assembly language
Not using interrupts, rather polling
Boots, runs printer, console (i/o) has 1 or two fixed tasks
1 very hard semester.
I would recommend reconsidering ignoring Unix systems services interface. If you use it you can get (and modify) utilities and applications.
Treating disk space as an array sounds simple, but you need to be able to name and retrieve files. This was a mistake in an early IBM 360 OS.
What format are you going to use for loadable executables?
Is device support built into the OS or are “loadable” drivers provided separately?
Suggest loadable drivers that have routines the OS calls to find if the hardware is present. What is your driver model? What are the operations supported?
There are probably a 100 questions you need to discover and decide upon before you put much effort into this.
/s/ Jim Williams
[one of the students referenced above
& an operating system architect of a “real-time” operating system for 10 years.]
Успехов!
ОЛОЛО ОЛОЛО ВЫ ВСЕ БЫДЛО И ХУЙЛО
you should make group of people who will work on your project.I also eagerly waiting for the invitations .so, plzz give reply ASAP
Well, yes, I think you’re crazy too, but you’re having fun and will learn a lot.
One niggle: as an alternative to Cygwin, you might want to look at MSYS or Gnuwin32 MSYS is part of the MinGW project. Cygwin is designed to provide Win32 ports of the GNU compiler suite and related tools, but does so by creating a POSIX compatibility layer encapsulated as a DLL, and linking against that. The good part is that a lot of *nix code builds “out of the box” under Cygwin. The bad part is that you must deal with *nix compatible PATH names and option delimiters, so mixing and matching Cygwin tools and native tools gets interesting.
MinGW attempts to provide the GNU compiler suite as a “native” Win32 application, and MSYS is a collection of the standard utilities to go with it, linked against Microsoft’s runtime. Gnuwin32 does likewise.
Since you’re using Visual Studio Express and MS’s compiler, native versions of the *nix tools might be preferable, and won’t have the overhead of Cygwin.
(If you absolutely *must* have a current version of Bash, Cygwin is your choice. Personally, I use an older [3.x] version of zsh when I want a *nix shell. But I mostly use things like grep, sed, awk, tr, head, tail, cut, less and the like, and all are readily available as native ports.)
______
Dennis
I’m currently trying out every one of these environments, but still find Cygwin the easiest.
I can’t find a decent way to build MinGW GCC cross-compiler.
I’ve built Clang-LLVM (it was quite easy with Visual Studio), but I can’t find a way to compile anything to machine code – it gives me asm files or LLVM bitcode files + LLVM launcher exes. Maybe I can compile these asm files with NASM? Or are they GNU ASM files, I can’t tell the difference.
It was plain easy to build a cross-compiler in Cygwin though. I’ve quite used to bash and *nix environment. So for now it’s my only option.
Using Windows and Visual Stuido to write an OS? WTF
Any other suggestions? 🙂
I’m currently trying out Eclipse with MinGW Cross compiler. But I still feel more comfortable with VS. And MSBuild can be configured with any compiler you like (not just MSVC, cl.exe and MASM). Look at this Android NDK integration: http://code.google.com/p/vs-android/, it uses GCC as a compiler. Or Check out Marmelade (multi platform mobile development kit): http://www.madewithmarmalade.com/ – it even runs an emulator after the build.
It really is just a matter of taste. You can do it in Vim, notepad or VS – it does not mater – what ever makes you feel comfortable and increases your development speed.
Key points:
1. Writing an OS is not writing a kernel.
2. Users don’t need OS, they need a lot of software.
>>Avoid legacy
Bad idea. In enterprise world everebody loves legacy.
>>How to write your own MBR and write it to Disk on windows.
Wait, you want to write a “modern” OS so forget about MBR – it’s a kind of deprecated.
>>Avoid the use of GRUB
What’s wrong with GRUB 2, seriously? Except “not invented here” argument 🙂
>>3rd gen Core i5 CPU with integrated Intel HD 4000 GPU
Intel GPU? Uh huh huh, this sucks. Without great support of NVIDIA/ATI cards most of PC users will not be interested in your OS.
My opinion: it is easy to write a programming-skilled AI before starting to write an OS alone. Think about it – you’re spending about 40-50 years to make it, the computation power grows during this time, and finally your AI will create an OS for you in five minutes 🙂
I hope you aren’t serious, just trolling.
Какого фига ты написал от имени православного z0mbie?
Gust Kaksis (aka gusc), you are have hard schizophrenia on your brain. You must be put itself of the stone to make the dead.
нда, лучше выучи язык, вместо того чтобы использоватса Google Tranlsate 🙂
Вы бы лучше рассказали, как там продвигается разработка Новой Революционной ОС. Постыдились бы хоть.
Уважаемый GusC. Судя по Вашей манере отвечать, хорошему знанию русского языка, нереальной амбициозности при нулевом уровне знаний об устройстве операционных систем, владению древним компьютером с установленной операционной системой Windows XP Вы ― недавно эммигрировавшийся в Латвию русский алгоколик и чтобы привлечь к себе внимание Вы в очередном припадке Вашей верной спутницы «белочки» и придумали совершенно самасбродную идею создания своей операционной системы. Вы ― посмешище для всех здравых людей. Вам в силу умопомешательства не понять, что несовмёстимую ни с чем операционную системӯ нужно обеспечивать драйверами для всех существующих современных устройств. Самый простой путь ― нанять команду из тысячи человек, которая будет заниматься портированием на Вашу ОС драйверов из Linux. Но Вам это не под силу ибо Вы загнётесь выплачивать миллионные кредиты, взятые для открытия фирмы и работы людей портирующих драйвера. И вы уже очень старый чтобы Вам начинать понимать азы ОСестроения. Примерно в 50 лет Вы только дойдёте до основ и поймёте что писать ОС на VC2010 Express это то же что писать её скажем на GW-BASIC и Вам таки придётся начать изучать ассемблер, который Вашему убитому белой горячкой мозгу просто «не по зубам». Советую Вам немедленно отказаться от своего проекта, иначе ещё ранее декабря следующего года Вы окажетесь в психиатрической клинике, будучи уже для всего мира полным посмешищем и «притчей во языцах». С наилучшими пожеланиями, Андрей Васильев.
For online craft supplies, UK shops have well organized and
user-friendly websites that will navigate you through proper channels.
You’re probably also hoping that this location is not far away and you don’t have to spend a lot of money to get there.
To achieve this level, you need to have your work recognized
publicly so that it can be purchased for its own consumption by museums and private collectors.
There are definitely lots of particulars like that to take into consideration.
That may be a nice level to carry up. I supply the thoughts above as
normal inspiration however clearly there are questions like the one you
carry up the place the most important thing
can be working in trustworthy good faith. I
don?t know if finest practices have emerged around issues like that, but
I’m positive that your job is clearly identified as a fair game. Each girls and boys really feel the influence of just a second’s
pleasure, for the remainder of their lives.
read any of the osdev pages lately? I have the same boot issue, hoping for a win32 PE EXE mode, but I cant find anonymous’ OS code for pascal win32.MAybe it was C, it hadnt taken off yet.
Very interesting! good luck
I was just searching for stuff that can help me on my way and i wanna say you are not crazy because I am planing to do that as well Thanks a lot for info also if you made something can you send me and example or give me an advice ?
u r great ! it’s my project too !!!!
I am also about to develop my own operating System. I’hv a basic knowledge of linux os and its operation.My configaration details is
1>Grub 2.0
2>Kernel 3.8
3>RPM Packeger
4>SHELL SCRIPT conf file
but at this moment what to do and which order.Thank you in advance .please Help me …… please…… help me
Am also doing such a project and am glad i have people who have similar views to mine, maybe we can discuss and share ideas
Hello – you might want to add the following to the list of reference materials in your article. It’s a nice listing of OS Dev tutorials: http://www.independent-software.com/writing-your-own-toy-operating-system/
https://github.com/honeydatax/dos
it very very crazy … LOL