CapROS Runtime Environment

This page provides a description of how the standard native CapROS runtime mechanism works. The CapROS kernel is not responsible for process construction; it is therefore not a party to any runtime conventions about these programs.

We first describe the runtime environment at the time the program's main() procedure is entered. Then we describe how the process is built and torn down.

1. Program Execution Environment

1.1 Standard Program Execution Environment

When the program's main() procedure is entered, the program has the following initial capabilities in its capability registers. Declarations of these symbols are in <domain/Runtime.h>.

KR_VOID	This key register (register zero) always holds a void key. This value is hard-wired, and cannot be changed by the program.
KR_CONSTIT	A read-only node key to the process's initial services (i.e. to the constituents node). The constituents node is the principal means of endowing a program with the specific capabilities it needs.
KR_KEYSTORE	A register reserved for a key to a supernode shared by all the processes ("threads") that share the same address space.
KR_CREATOR	A start key to the process's process creator, with the precludeDestroy restriction. A process can use this to tear itself down.
KR_SELF	A process key to the process itself.
KR_BANK	A key to the process's space bank.
KR_SCHED	A duplicate of the process's schedule key.
KR_RETURN	A resume key to the process that caused this process to be started.

The process is free to use other key registers for its own purposes. You should not assume these registers are initially void; they may have keys left over from constructing the process. There are names and conventions for these key registers:

KR_APP(n)	KR_APP(0), KR_APP(1), etc. are a set of key registers typically used for long-term storage of keys. Currently n can be up to 16, but it would be unwise to count on more than about 14.
KR_TEMP0, KR_TEMP1, KR_TEMP2, KR_TEMP3	Key registers for very short-term use. The convention is that these should be used only between procedure calls, or as parameters or results of procedure calls. In other words, unless otherwise documented, you must assume that any procedure you call may disturb these registers. (IDL procedures (for example capros_Node_getSlot()) are a notable exception; they do not disturb any key registers except those explicitly specified as parameters.)
KR_ARG(n)	KR_ARG(0), KR_ARG(1), and KR_ARG(2) are key registers that may be used by the program for any purpose. Often they are used by a server to receive keys from the caller. When a constructor requestor object is called to construct a new instance, the new process receives the third key parameter to the request in KR_ARG(0).

The process initially has no keeper. Perhaps in the future there will be a way to define an initial keeper.

Your main procedure should be declared as simply int main(void). CapROS programs don't receive parameters via argc and argv.

When the program is finished, it should destroy itself and if appropriate return to its caller (as described below), rather than return from main(). If the program does return from main(). the return code is discarded and the process is not torn down.

1.2 Linux Program Execution Environment

Programs that run in the Linux kernel emulation environment have additional conventions. For example, some of the KR_APP(n) registers are used by the emulation. See the header file <linuxk/lsync.h> for details.

1.3 Memory Allocation

There are two models of memory allocation used in CapROS. The initialization and teardown procedures are different for the two models.

1.3.1 Automatic Memory Allocation

In this model, storage for bss, data, stack, and heap is automatically allocated from a space bank when the storage is first referenced. This convenience comes at a cost; the address space consists of a VCS (Virtual Copy Segment) that gets control on page faults and uses copy-on-write techniques to create the storage on demand.

There is one important caveat when using this model. If the process issues a capability invocation that receives a data string into memory, that memory cannot be automatically allocated. (No page fault occurs, because the string is transferred under the control of the sending process.) If not previously allocated, the invocation will fail. Before receiving a string into memory, you must ensure that the memory is allocated by storing to each page in the string.

1.3.2 Self-managed Memory Allocation

To minimize overhead, some programs may find it more appropriate to manage their own memory. This is actually necessary to implement the richer environment. Low overhead comes at the cost of correspondingly reduced runtime support.

In this model, storage for bss, data, and stack are allocated by the runtime system when the process is initialized (this is described below). If the process uses a heap, it must define its own sbrk() or _sbrk() procedures to manage heap storage. Most programs using this model don't require a heap.

The stack does not grow, so the program must know in advance the amount of stack required. Since program development tools seldom provide this information, the stack size must be conservatively estimated or determined experimentally.

2. Process Initialization

There are several variants of CapROS process initialization, to accommodate programs with various requirements. Most programmers will not need to know this detail. It is documented here mainly because the information is hard to find elsewhere.

2.1 Program linking

A basic understanding of how a CapROS program is linked into an executable file is needed to understand initialization. CapROS programs are linked using a cross-linker with some custom features. Look at the CapROS source code for some examples of how to use the cross-linker including required libraries.

The cross-linker automatically includes crt1.o which contains the first code executed in the program. Execution begins at the symbol _start and does the following:

1. Examine the variable __rt_runtime_hook and if it is nonzero, branch to that address. We'll see below how this is used.
2. Load the stack pointer with the contents of the variable __rt_stack_pointer.
3. Call the procedure __domain_startup.
4. __domain_startup is responsible for initializing and finalizing the language environment. This is where C++ static constructors and destructors are called. This is done by calling atexit(_fini) and _init(), and then calling exit ater main has returned. Implementation issue: This logic is currently disabled, because atexit pulls in malloc and sbrk, which must be avoided in certain very low-level processes. Consequently C++ is not currently supported.
5. __domain_startup calls main.
6. In most cases main will never return. When the program is finished, it should destroy itself and if appropriate return to its caller (as described below). If it does, __domain_startup passes the return code to _exit (rather than exit).
7. _exit performs an invocation on the process's own key (which must be in KR_SELF) that causes the process to stop running. The return code is discarded and the process is not torn down. _exit never returns.

The cross-linker also automatically includes crti.o and crtn.o which are used to build the _init and _fini procedures in the .init and .fini sections respectively.

2.2 Linkage Models

2.2.1 Small Linkage Model

The option -small-space instructs the linker to build a "small" address space. This means that the default locations for the program, stack, and heap are chosen to keep the address space compact, which improves efficiency. In particular the address space can be represented in a single GPT. This option causes the linker to use the library libcapros-small.

libcapros-small defines __rt_runtime_hook to point to a procedure that currently does the following:

1. If the variable __rt_stack_pages is nonzero, it specifies the number of pages of stack to create. The address space is cloned into a new GPT, the stack pages are allocated and installed in the new GPT at addresses 0x1f000 and below, and the new GPT is installed as the process's address space. (This is tricky assembler code, because the process starts out with no read-write storage and must bootstrap itself.)

   In this case the variable __rt_stack_pointer must contain 0x20000. On the other hand, if __rt_stack_pages is zero, the stack must have been created already and __rt_stack_pointer must point to its bottom.

   If you do not define __rt_stack_pages, the library defines it to contain 1.

2. The stack pointer is set to the value in __rt_stack_pointer.

   If you do not define __rt_stack_pointer, the library defines it to contain 0x20000.

3. Pages for the .bss and .data sections are allocated and installed in the address space, and the .bss area is initialized. The .data area will be initially zero.
4. We then go back to _start, after setting __rt_runtime_hook to zero so this procedure won't be repeated.

This procedure will probably change at some point.

If you define __rt_runtime_hook, the linker will not load the definition in libcapros-small, so you can specify your own procedure (or none) instead of the above. You had better know what you are doing.

2.2.2 Large Linkage Model

The option -large-space instructs the linker to build a "large" address space. This means that the default locations for the program, stack, and heap are chosen to maximize the space available for stack and heap. This option causes the linker to use the library option -lcapros-large. This option is the default.

Currently, this option weakly defines __rt_runtime_hook as zero. The address space is assumed to be completely constructed before the process runs. __rt_stack_pointer must contain the initial stack pointer. libcapros-large defines __rt_stack_pointer to be 0x2000000 on ARM and 0x80000000 on x86; you can override this by defining it yourself.

2.2.3 Small Address Spaces

On the IA32 architecture, the kernel currently has an optimization for address spaces that fit in the range [0, 0x20000). Processes with such address spaces enjoy better context switch efficiency.

On the ARM architecture, the kernel has an optimization for address spaces that fit in the range [0, 0x2000000). Processes with such address spaces enjoy much better context switch efficiency, because the kernel uses the ARM Fast Context Switch Extension.

2.3 Initialization using the Constructor

CapROS programs are usually built by Constructors. The first step in process construction is to allocate the new process and initialize its key registers to the default runtime environment defined above.

The Constructor supports two types of address spaces.

• If the address space key was specified with the Constructor.insertVCSAddressSpace32 method, the specified address space key is a constructor for an address space. To avoid having the Constructor busy longer than necessary, the constructor for the new process's address space is called by the process itself. To accomplish this bootstrapping, the process first executes a temporary program known as the protospace. The protospace invokes the constructor to get an address space. That address space is then installed as the process's address space, and the new program is started. Usually, the new address space constructor is a VCS, and the stack and data areas will be filled in as page faults occur.

• If the address space key was specified with the Constructor.insertAddressSpace32 method, the specified address space key is installed directly as the process's address space, and the new program is started. In this case the program has no per-instance storage and is responsible for any further population of its address space. There are two common ways to do so:

  • If the new program was linked as a small space, it can create its own stack and data areas as described above using the small-space runtime hook.
  • To support construction of more complex address spaces, there is a means to build a read-only immutable program that can then build a more general address space. This uses an assembly program that interprets a table of invocations to be executed to build the address space. You must define the table to build the space as you want it. For details see base/lib/constructor/InterpreterTable.h and related files.

3. Process Teardown

Just as the process must arrange to initialize itself, it must arrange to destroy itself. Since the process created its own address space, it must destroy it. The challenge of destruction is the capros_key_destroy request. Careful look at the object reference manual will show that the destroy operation is expected to return. The question is, "How can a program that no longer has an address space return to anyone?" This is the inverse of the bootstrapping problem.

The answer is, we rely on the process creator for help. A process that wishes to support the capros_key_destroy method must hang on to its process creator key. If the process runs in a kept address space, it must also retain access to the protospace address space.

When asked to destroy itself, the process should of course first destroy anything it has explicitly built. Then, how teardown is accomplished depends on the process address space.

• If it was built as a small process by the small runtime hook, it must undo that by calling protospace_destroy_small. protospace_destroy_small is declared and documented in domain/ProtoSpaceDS.h.

  protospace_destroy_small installs the protospace address space and goes to an entry point in that space known as the telospace. (Greek root telo- is the opposite of proto-.) That code returns to the space bank any pages in the old address space that were created, returns the GPT to the bank, and then invokes the destroyCallerAndReturn method on the process creator.

• If the process's address space was built from a VCS using the Constructor protospace, it can call protospace_destroy. However, there is a note that that procedure is obsolete. This may not be working currently.

• If the address space was a more complex space built using the interpreter facility, a corresponding facility will tear down the space. See InterpreterDestroy, which is declared and documented in base/lib/constructor/InterpreterTable.h.

Note that all this is initiated by the program (usually in response to a destroy request), which means it happens before main() returns. Since main() never returns, static destructors and atexit() procedures are never called. These facilities are not supported.

Note that protospace itself is a shared resource that does not get destroyed, as is the interpreter table.