Caution

This is documentation for the development version of the project, aka master branch. If you installed Gramine from packages, documentation for the stable version is available at https://gramine.readthedocs.io/en/stable/.

PAL host ABI

TODO: This document is outdated and needs a proper review.

PAL Host ABI is the interface used by Gramine to interact with its host. It is translated into the host’s native ABI (e.g. system calls for UNIX) by a layer called the Platform Adaptation Layer (PAL). A PAL not only exports a set of APIs (PAL APIs) that can be called by the library OS, but also acts as the loader that bootstraps the library OS. The design of PAL Host ABI strictly follows three primary principles, to guarantee functionality, security, and portability:

  • The host ABI must be stateless.
  • The host ABI must be a narrowed interface to reduce the attack surface.
  • The host ABI must be generic and independent from the native ABI of any of the supported hosts.

Most of the PAL Host ABI is adapted from the Drawbridge library OS.

PAL as loader

Regardless of the actual implementation, we require PAL to be able to load ELF-format binaries as executables or dynamic libraries, and perform the necessary dynamic relocation. PAL needs to look up all unresolved symbols in loaded binaries and resolve the ones matching the names of PAL APIs. PAL does not and will not resolve other unresolved symbols, so the loaded libraries and executables must resolve them afterwards.

After loading the binaries, PAL needs to load and interpret the manifest files. The manifest syntax is described in Manifest syntax.

Manifest and executable loading

To run a program in Gramine the PAL loader needs a manifest, which will describe the whole environment inside Gramine namespace. It also describes which executable to start first (via libos.entrypoint).

Data types and variables

Data types

PAL handles

PAL_HANDLE is the type of identifiers that are returned by PAL when opening or creating resources. It is an opaque type, that should not be accessed outside of PAL and its details depend on the actual PAL version (host). There is a common header (present on all PAL hosts) accessible from PAL code, which allows for checking the handle type:

typedef struct {
    struct {
        PAL_IDX type;
    } hdr;
    /* other host-specific definitions */
}* PAL_HANDLE;

Rest of the fields are private to specific PAL hosts, but they usually define different handle subtypes that represent different resources such as files, directories, pipes or sockets. The actual memory allocated for the PAL handles may be variable-sized.

Basic types

typedef uint32_t PAL_IDX

an index

PAL public state

All PALs in Gramine expose a structure that provides static immutable information about the current process and its host. The address of the structure can be retrieved via PalGetPalPublicState() and can be memorized in a global variable for ease of use.

struct pal_public_state

Public Members

toml_table_t* manifest_root

program manifest

PAL_HANDLE parent_process

handle of parent process

PAL_HANDLE first_thread

handle of first thread

int log_level

what log messages to enable

bool disable_aslr

disable ASLR (may be necessary for restricted environments)

void* user_address_start

User address range start

void* user_address_end

User address range end

struct pal_public_state::@10* preloaded_ranges

array of memory ranges which are preoccupied

size_t alloc_align

Host allocation alignment.

This currently is (and most likely will always be) indistinguishable from the page size, looking from the LibOS perspective. The two values can be different on the PAL level though, see e.g. SYSTEM_INFO::dwAllocationGranularity on Windows.

PAL public state - topology information

struct pal_cpu_info
struct pal_topo_info

PAL APIs

The PAL APIs contain a number of functions that can be called from the library OS.

Memory allocation

The ABI includes three calls to allocate, free, and modify the permission bits on page-base virtual memory. Permissions include read, write, execute, and guard. Memory regions can be unallocated, reserved, or backed by committed memory.

int PalVirtualMemoryAlloc(void ** addr_ptr, size_t size, pal_alloc_flags_t alloc_type, pal_prot_flags_t prot)

Allocate virtual memory for the library OS and zero it out.

*addr_ptr can be any valid address aligned at the allocation alignment or NULL, in which case a suitable address will be picked automatically. Any memory previously allocated at the same address will be discarded (only if *addr_ptr was provided). Overwriting any part of PAL memory is forbidden. On successful return *addr_ptr will contain the allocated address (which can differ only in the NULL case).

Parameters
  • addr_ptr: *addr_ptr should contain requested address or NULL. On success, it will be set to the allocated address.
  • size: Must be a positive number, aligned at the allocation alignment.
  • alloc_type: A combination of any of the PAL_ALLOC_* flags.
  • prot: A combination of the PAL_PROT_* flags.

int PalVirtualMemoryFree(void * addr, size_t size)

Deallocate a previously allocated memory mapping.

Both

addr and size must be non-zero and aligned at the allocation alignment.
Parameters
  • addr: The address.
  • size: The size.

typedef uint32_t pal_alloc_flags_t

memory allocation flags

typedef uint32_t pal_prot_flags_t

memory protection flags

int PalVirtualMemoryProtect(void * addr, size_t size, pal_prot_flags_t prot)

Modify the permissions of a previously allocated memory mapping.

Both

addr and size must be non-zero and aligned at the allocation alignment.
Parameters

Process creation

The ABI includes one call to create a child process and one call to terminate the running process. A child process does not inherit any objects or memory from its parent process and the parent process may not modify the execution of its children. A parent can wait for a child to exit using its handle. Parent and child may communicate through I/O streams provided by the parent to the child at creation.

int PalProcessCreate(const char ** args, PAL_HANDLE * handle)

Create a new process.

Loads and executes the same binary as currently executed one (

loader.entrypoint), and passes the new arguments.
Parameters
  • args: An array of strings the arguments to be passed to the new process.
  • handle: On success contains the process handle.

TODO: args is only used by PAL regression tests, and should be removed at some point.

void PalProcessExit(int exit_code)

Terminate all threads in the process immediately.

Parameters
  • exit_code: The exit value returned to the host.

Stream creation/connect/open

The stream ABI includes nine calls to open, read, write, map, unmap, truncate, flush, delete and wait for I/O streams and three calls to access metadata about an I/O stream. The ABI purposefully does not provide an ioctl call. Supported URI schemes include: file:, pipe:, http:, https:, tcp:, udp:, pipe.srv:, http.srv, tcp.srv: and udp.srv:. The latter four schemes are used to open inbound I/O streams for server applications.

int PalStreamOpen(const char * uri, enum pal_access access, pal_share_flags_t share_flags, enum pal_create_mode create, pal_stream_options_t options, PAL_HANDLE * handle)

Open/create a stream resource specified by uri.

Supported URI types:

  • file:..., dir:...: Files or directories on the host file system. If PAL_CREATE_TRY or PAL_CREATE_ALWAYS is given in create flags, the file/directory will be created.
  • dev:...: Open a device as a stream. For example, dev:tty represents the standard I/O.
  • pipe.srv:<name>, pipe:<name>, pipe:: Open a byte stream that can be used for RPC between processes. The server side of a pipe can accept any number of connections. If pipe: is given as the URI (i.e., without a name), it will open an anonymous bidirectional pipe.
Return
0 on success, negative error code on failure.
Parameters
  • uri: The URI of the stream to be opened/created.
  • access: See pal_access.
  • share_flags: A combination of the PAL_SHARE_* flags.
  • create: See pal_create_mode.
  • options: A combination of the PAL_OPTION_* flags.
  • handle[out]: If the resource is successfully opened or created, a PAL handle is returned in *handle for further access such as reading or writing.

int PalStreamWaitForClient(PAL_HANDLE handle, PAL_HANDLE * client, pal_stream_options_t options)

Block until a new connection is accepted and return the PAL handle for the connection.

This API is only available for handles that are opened with

pipe.srv:....
Parameters
  • handle: Handle to accept a new connection on.
  • client: On success holds handle for the new connection.
  • options: Flags to set on client handle.

int PalStreamRead(PAL_HANDLE handle, uint64_t offset, size_t * count, void * buffer)

Read data from an open stream.

If

handle is a directory, PalStreamRead fills the buffer with the null-terminated names of the directory entries.
Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the stream.
  • offset: Offset to read at. If handle is a file, offset must be specified at each call.
  • count: Contains size of buffer. On success, will be set to the number of bytes read.
  • buffer: Pointer to the buffer to read into.

int PalStreamWrite(PAL_HANDLE handle, uint64_t offset, size_t * count, void * buffer)

Write data to an open stream.

Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the stream.
  • offset: Offset to write to. If handle is a file, offset must be specified at each call.
  • count: Contains size of buffer. On success, will be set to the number of bytes written.
  • buffer: Pointer to the buffer to write from.

int PalStreamDelete(PAL_HANDLE handle, enum pal_delete_mode delete_mode)

Delete files or directories on the host or shut down the connection of TCP/UDP sockets.

Parameters

int PalStreamMap(PAL_HANDLE handle, void ** addr_ptr, pal_prot_flags_t prot, uint64_t offset, size_t size)

Map a file to a virtual memory address in the current process.

Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the stream to be mapped.
  • addr_ptr: See PalVirtualMemoryAlloc.
  • prot: See PalVirtualMemoryAlloc.
  • offset: Offset in the stream to be mapped. Must be properly aligned.
  • size: Size of the requested mapping. Must be non-zero and properly aligned.

int PalStreamUnmap(void * addr, size_t size)

Unmap virtual memory that is backed by a file stream.

addr and size must be aligned at the allocation alignment.

Return
0 on success, negative error code on failure.

int PalStreamSetLength(PAL_HANDLE handle, uint64_t length)

Set the length of the file referenced by handle to length.

Return
0 on success, negative error code on failure.

int PalStreamFlush(PAL_HANDLE handle)

Flush the buffer of a file stream.

Return
0 on success, negative error code on failure.

int PalSendHandle(PAL_HANDLE target_process, PAL_HANDLE cargo)

Send a PAL handle to a process.

Return
0 on success, negative error code on failure.
Parameters
  • target_process: The handle to the target process where cargo will be sent.
  • cargo: The handle to send.

int PalReceiveHandle(PAL_HANDLE source_process, PAL_HANDLE * out_cargo)

Receive a handle from another process.

Return
0 on success, negative error code on failure.
Parameters
  • source_process: The handle to the source process from which cargo will be received.
  • out_cargo: The received handle.

int PalStreamAttributesQuery(const char * uri, PAL_STREAM_ATTR * attr)

Query the attributes of a named stream.

This API only applies for URIs such as file:..., dir:..., and dev:....

typedef struct _PAL_STREAM_ATTR PAL_STREAM_ATTR
struct _PAL_STREAM_ATTR
int PalStreamAttributesQueryByHandle(PAL_HANDLE handle, PAL_STREAM_ATTR * attr)

Query the attributes of an open stream.

This API applies to any stream handle.

int PalStreamAttributesSetByHandle(PAL_HANDLE handle, PAL_STREAM_ATTR * attr)

Set the attributes of an open stream.

Calling this function on the same handle concurrently is not allowed (i.e. callers must ensure mutual exclusion).

int PalStreamChangeName(PAL_HANDLE handle, const char * uri)

This API changes the name of an open stream.

Flags used for stream manipulation

pal_access

Stream Access Flags

Values:

PAL_ACCESS_RDONLY
PAL_ACCESS_WRONLY
PAL_ACCESS_RDWR
PAL_ACCESS_BOUND
typedef uint32_t pal_share_flags_t

stream sharing flags

pal_create_mode

stream create mode

Values:

PAL_CREATE_NEVER

Fail if file does not exist

PAL_CREATE_TRY

Create file if file does not exist

PAL_CREATE_ALWAYS

Create file and fail if file already exists

PAL_CREATE_IGNORED

Magic value for calls to handle types which ignore creation mode

typedef uint32_t pal_stream_options_t

stream misc flags

pal_delete_mode

Values:

PAL_DELETE_ALL

delete the whole resource / shut down both directions

PAL_DELETE_READ

shut down the read side only

PAL_DELETE_WRITE

shut down the write side only

typedef uint32_t pal_wait_flags_t

Socket handling

pal_socket_domain

Values:

PAL_DISCONNECT
PAL_IPV4
PAL_IPV6
pal_socket_type

Values:

PAL_SOCKET_TCP
PAL_SOCKET_UDP
struct pal_socket_addr
struct pal_iovec
int PalSocketCreate(enum pal_socket_domain domain, enum pal_socket_type type, pal_stream_options_t options, PAL_HANDLE * out_handle)

Create a socket handle.

Return
0 on success, negative error code on failure.
Parameters
  • domain: Domain of the socket.
  • type: Type of the socket.
  • options: Flags to set on the handle.
  • out_handle: On success contains the socket handle.

int PalSocketBind(PAL_HANDLE handle, struct pal_socket_addr * addr)

Bind a socket to a local address.

Can be called only once per socket.

Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket.
  • addr: Address to bind to. If the protocol allows for some ephemeral data (e.g. port 0 in IPv4), it will be overwritten to the actual data used.

int PalSocketListen(PAL_HANDLE handle, unsigned int backlog)

Turn a socket into a listening one.

Can be called multiple times, to change

backlog.
Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket.
  • backlog: Size of the pending connections queue.

int PalSocketAccept(PAL_HANDLE handle, pal_stream_options_t options, PAL_HANDLE * out_client, struct pal_socket_addr * out_client_addr, struct pal_socket_addr * out_local_addr)

Accept a new connection on a socket.

This function can be safely called concurrently.

Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket. Must be in listening mode.
  • options: Flags to set on the new handle.
  • out_client: On success contains a handle for the new connection.
  • out_client_addr: On success contains the remote address of the new connection. Can be NULL, to ignore the result.
  • out_local_addr: On success contains the local address of the new connection. Can be NULL, to ignore the result.

int PalSocketConnect(PAL_HANDLE handle, struct pal_socket_addr * addr, struct pal_socket_addr * out_local_addr)

Connect a socket to a remote address.

Can also be used to disconnect the socket, if #PAL_DISCONNECT is passed in

addr.
Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket.
  • addr: Address to connect to.
  • out_local_addr: On success contains the local address of the socket. Can be NULL, to ignore the result.

int PalSocketSend(PAL_HANDLE handle, struct pal_iovec * iov, size_t iov_len, size_t * out_size, struct pal_socket_addr * addr, bool force_nonblocking)

Send data.

Data is sent atomically, i.e. data from two

PalSocketSend calls will not be interleaved.
Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket.
  • iov: Array of buffers with data to send.
  • iov_len: Length of iov array.
  • out_size: On success contains the number of bytes sent.
  • addr: Destination address. Can be NULL if the socket was connected.
  • force_nonblocking: If true this request should not block. Otherwise just use whatever mode the handle is in.

int PalSocketRecv(PAL_HANDLE handle, struct pal_iovec * iov, size_t iov_len, size_t * out_total_size, struct pal_socket_addr * addr, bool force_nonblocking)

Receive data.

Data is received atomically, i.e. data from two

PalSocketRecv calls will not be interleaved.
Return
0 on success, negative error code on failure.
Parameters
  • handle: Handle to the socket.
  • iov: Array of buffers for received data.
  • iov_len: Length of iov array.
  • out_total_size: On success contains the number of bytes received (TCP) or the size of the packet (UDP), which might be greater than the total size of buffers in iov array.
  • addr: Source address. Can be NULL to ignore the source address.
  • force_nonblocking: If true this request should not block. Otherwise just use whatever mode the handle is in.

Thread creation

The ABI supports multithreading through five calls to create, sleep, yield the scheduler quantum for, resume execution of, and terminate threads, as well as seven calls to create, signal, and block on synchronization objects.

int PalThreadCreate(int(*callback)(void *), void * param, PAL_HANDLE * handle)

Create a thread in the current process.

Parameters
  • addr: Address of an entry point of execution for the new thread.
  • param: Pointer argument that is passed to the new thread.
  • handle: On success contains the thread handle.

void PalThreadYieldExecution(void)

Yield the current thread such that the host scheduler can reschedule it.

void PalThreadExit(int * clear_child_tid)

Terminate the current thread.

Parameters
  • clear_child_tid: Pointer to memory that is erased on thread exit to notify LibOS (which in turn notifies the parent thread if any); if clear_child_tid is NULL, then PAL doesn’t do the clearing.

int PalThreadResume(PAL_HANDLE thread)

Resume a thread.

Exception handling

pal_event

Values:

PAL_EVENT_NO_EVENT

pseudo event, used in some APIs to denote a lack of event

PAL_EVENT_ARITHMETIC_ERROR

arithmetic error (div-by-zero, floating point exception, etc.)

PAL_EVENT_MEMFAULT

segmentation fault, protection fault, bus fault

PAL_EVENT_ILLEGAL

illegal instructions

PAL_EVENT_QUIT

terminated by external program (see “sys.enable_sigterm_injection” manifest option)

PAL_EVENT_INTERRUPTED

interrupted (usually internally to handle aync event)

PAL_EVENT_NUM_BOUND
typedef struct PAL_CONTEXT PAL_CONTEXT
struct PAL_CONTEXT
typedef void(* pal_event_handler_t)(bool is_in_pal, uintptr_t addr, PAL_CONTEXT *context)

Type of exception handlers (upcalls).

Parameters
  • is_in_pal: true if the exception happened inside PAL.
  • addr: Address of the exception (meaningful only for sync exceptions).
  • context: CPU context at the moment of exception.

void PalSetExceptionHandler(pal_event_handler_t handler, enum pal_event event)

Set the handler for the specific exception event.

Parameters

Synchronization

int PalEventCreate(PAL_HANDLE * handle, bool init_signaled, bool auto_clear)

Create an event handle.

Creates a handle to an event that resembles WinAPI synchronization events. A thread can set (signal) the event using

PalEventSet, clear (unset) it using PalEventClear or wait until the event becomes set (signaled) using PalEventWait.
Parameters
  • handle: On success *handle contains pointer to the event handle.
  • init_signaled: Initial state of the event (true - set, false - not set).
  • auto_clear: true if a successful wait for the event should also reset (consume) it.

void PalEventSet(PAL_HANDLE handle)

Set (signal) an event.

If the event is already set, does nothing.

This function has release semantics and synchronizes with PalEventWait.

void PalEventClear(PAL_HANDLE handle)

Clear (unset) an event.

If the event is not set, does nothing.

int PalEventWait(PAL_HANDLE handle, uint64_t * timeout_us)

Wait for an event handle.

timeout_us points to a value that specifies the maximal time (in microseconds) that this function should sleep if this event is not signaled in the meantime. Specifying NULL blocks indefinitely. Note that in any case this function can return earlier, e.g. if a signal has arrived, but this will be indicated by the returned error code. After returning (both successful and not), timeout_us will contain the remaining time (time that need to pass before we hit original timeout_us).

Return
0 if the event was triggered, negative error code otherwise (#PAL_ERROR_TRYAGAIN in case of timeout triggering)
Parameters
  • handle: Handle to wait on, must be of “event” type.
  • timeout_us: Timeout for the wait.

This function has acquire semantics and synchronizes with PalEventSet.

Objects

int PalStreamsWaitEvents(size_t count, PAL_HANDLE * handle_array, pal_wait_flags_t * events, pal_wait_flags_t * ret_events, uint64_t * timeout_us)

Poll - wait for an event to happen on at least one handle.

timeout_us contains remaining timeout both on successful and failed calls.

Return
0 if there was an event on at least one handle, negative error code otherwise.
Parameters
  • count: The number of items in handle_array.
  • handle_array: Array of handles to poll.
  • events: Requested events for each handle.
  • ret_events: Events that were detected on each handle.
  • timeout_us: Timeout for the wait (NULL to block indefinitely).

void PalObjectClose(PAL_HANDLE object_handle)

Close (deallocate) a PAL handle.

Miscellaneous

The ABI includes seven assorted calls to get wall clock time, generate cryptographically-strong random bits, flush portions of instruction caches, increment and decrement the reference counts on objects shared between threads, and to obtain an attestation report and quote.

int PalDebugLog(const void * buffer, size_t size)

Output a message to the debug stream.

Return
0 on success, negative error code on failure.
Parameters
  • buffer: Message to write.
  • size: buffer size.

struct pal_public_state* PalGetPalPublicState(void)
int PalSystemTimeQuery(uint64_t * time)

Get the current time.

Parameters
  • time: On success holds the current time in microseconds.

int PalRandomBitsRead(void * buffer, size_t size)

Cryptographically secure RNG.

Return
0 on success, negative on failure.
Parameters
  • buffer: Output buffer.
  • size: buffer size.

int PalSegmentBaseGet(enum pal_segment_reg reg, uintptr_t * addr)

Get segment register base.

Return
0 on success, negative error value on failure.
Parameters
  • reg: The register base to get (#pal_segment_reg).
  • addr: The address where result will be stored.

int PalSegmentBaseSet(enum pal_segment_reg reg, uintptr_t addr)

Set segment register.

Return
0 on success, negative error value on failure.
Parameters
  • reg: The register base to be set (#pal_segment_reg).
  • addr: The address to be set.

pal_segment_reg

Values:

PAL_SEGMENT_FS
PAL_SEGMENT_GS
size_t PalMemoryAvailableQuota(void)

Return the amount of currently available memory for LibOS/application usage.

int PalCpuIdRetrieve(uint32_t leaf, uint32_t subleaf, uint32_t values[CPUID_WORD_NUM])

Return CPUID information, based on the leaf/subleaf.

Parameters
  • values: The array of the results.

int PalAttestationReport(const void * user_report_data, size_t * user_report_data_size, void * target_info, size_t * target_info_size, void * report, size_t * report_size)

Obtain the attestation report (local) with user_report_data embedded into it.

Currently works only for Linux-SGX PAL, where

user_report_data is a blob of exactly 64B, target_info is an SGX target_info struct of exactly 512B, and report is an SGX report obtained via the EREPORT instruction (exactly 432B). If target_info contains all zeros, then this function additionally returns this enclave’s target info in target_info. Useful for local attestation.
Parameters
  • user_report_data: Report data with arbitrary contents (typically uniquely identifies this Gramine instance). Must be a 64B buffer in case of SGX PAL.
  • user_report_data_size: Caller specifies size of user_report_data; on return, contains PAL-enforced size of user_report_data (64B in case of SGX PAL).
  • target_info: Target info of target enclave for attestation. If it contains all zeros, it is populated with this enclave’s target info. Must be a 512B buffer in case of SGX PAL.
  • target_info_size: Caller specifies size of target_info; on return, contains PAL-enforced size of target_info (512B in case of SGX PAL).
  • report: Attestation report with user_report_data embedded, targeted for an enclave with provided target_info. Must be a 432B buffer in case of SGX PAL.
  • report_size: Caller specifies size of report; on return, contains PAL-enforced size of report (432B in case of SGX PAL).

The caller may specify *user_report_data_size, *target_info_size, and *report_size as 0 and other fields as NULL to get PAL-enforced sizes of these three structs.

int PalAttestationQuote(const void * user_report_data, size_t user_report_data_size, void * quote, size_t * quote_size)

Obtain the attestation quote with user_report_data embedded into it.

Currently works only for Linux-SGX PAL, where

user_report_data is a blob of exactly 64B and quote is an SGX quote obtained from Quoting Enclave via AESM service.
Parameters
  • user_report_data: Report data with arbitrary contents (typically uniquely identifies this Gramine instance). Must be a 64B buffer in case of SGX PAL.
  • user_report_data_size: Size in bytes of user_report_data. Must be exactly 64B in case of SGX PAL.
  • quote: Attestation quote with user_report_data embedded.
  • quote_size: Caller specifies maximum size allocated for quote; on return, contains actual size of obtained quote.

int PalGetSpecialKey(const char * name, void * key, size_t * key_size)

Get special key (specific to PAL host).

Retrieve the value of a special key. Currently implemented for Linux-SGX PAL, which supports two such keys:

_sgx_mrenclave and _sgx_mrsigner.
Parameters
  • name: Key name.
  • key: On success, will be set to retrieved key.
  • key_size: Caller specifies maximum size for key. On success, will contain actual size.

If a given key is not supported by the current PAL host, the function will return -PAL_ERROR_NOTIMPLEMENTED.