PAL host ABI¶
PAL Host ABI is the interface used by Gramine to interact with its host. It is translated into the host’s native ABI (e.g. system calls for UNIX) by a layer called the Platform Adaptation Layer (PAL). A PAL not only exports a set of APIs (PAL APIs) that can be called by the library OS, but also acts as the loader that bootstraps the library OS. The design of PAL Host ABI strictly follows three primary principles, to guarantee functionality, security, and portability:
- The host ABI must be stateless.
- The host ABI must be a narrowed interface to reduce the attack surface.
- The host ABI must be generic and independent from the native ABI of any of the supported hosts.
Most of the PAL Host ABI is adapted from the Drawbridge library OS.
PAL as loader¶
Regardless of the actual implementation, we require PAL to be able to load ELF-format binaries as executables or dynamic libraries, and perform the necessary dynamic relocation. PAL needs to look up all unresolved symbols in loaded binaries and resolve the ones matching the names of PAL APIs. PAL does not and will not resolve other unresolved symbols, so the loaded libraries and executables must resolve them afterwards.
After loading the binaries, PAL needs to load and interpret the manifest files. The manifest syntax is described in Manifest syntax.
Manifest and executable loading¶
To run a program in Gramine the PAL loader needs a manifest, which will
describe the whole environment inside Gramine namespace. It also describes
which executable to start first (via libos.entrypoint
).
Data types and variables¶
Data types¶
PAL handles¶
The PAL handles are identifiers that are returned by PAL when opening or creating resources. The basic data structure of a PAL handle is defined as follows:
typedef union pal_handle {
struct {
PAL_IDX type;
} hdr;
/* other resource-specific definitions */
}* PAL_HANDLE;
-
union
pal_handle
¶
-
typedef union pal_handle*
PAL_HANDLE
¶
As shown above, a PAL handle is usually defined as a union data type that contains different subtypes that represent each resource such as files, directories, pipes or sockets. The actual memory allocated for the PAL handles may be variable-sized.
Basic types¶
-
typedef uint64_t
PAL_NUM
¶ a number
-
typedef uint32_t
PAL_FLG
¶ a set of flags
-
typedef void*
PAL_PTR
¶ a pointer to memory or buffer (something other than string)
-
typedef const char*
PAL_STR
¶ a pointer to a C-string
-
typedef uint32_t
PAL_IDX
¶ an index
-
typedef bool
PAL_BOL
¶ a boolean value (either PAL_TRUE or PAL_FALSE)
This data type is commonly used as the return value of a PAL API to determine whether the call succeeded
-
typedef struct PAL_PTR_RANGE_
PAL_PTR_RANGE
¶
-
struct
PAL_PTR_RANGE_
¶
Gramine control block¶
The control block in Gramine is a structure that provides static immutable
information about the current process and its host. The address of the control
block is resolved via DkGetPalControl()
and can be memorized in a global
variable for ease of use.
The fields of the Gramine control block are defined as follows:
-
typedef struct PAL_CONTROL_
PAL_CONTROL
¶
-
struct
PAL_CONTROL_
¶ Public Members
-
toml_table_t*
manifest_root
¶ program manifest
-
PAL_HANDLE
parent_process
¶ handle of parent process
-
PAL_HANDLE
first_thread
¶ handle of first thread
-
int
log_level
¶ what log messages to enable
-
PAL_PTR_RANGE
user_address
¶ The range of user addresses
-
struct PAL_CONTROL_::@9*
preloaded_ranges
¶ array of memory ranges which are preoccupied
-
PAL_NUM
alloc_align
¶ Host allocation alignment.
This currently is (and most likely will always be) indistinguishable from the page size, looking from the LibOS perspective. The two values can be different on the PAL level though, see e.g. SYSTEM_INFO::dwAllocationGranularity on Windows.
-
PAL_CPU_INFO
cpu_info
¶ CPU information (only required ones)
-
PAL_MEM_INFO
mem_info
¶ memory information (only required ones)
-
PAL_TOPO_INFO
topo_info
¶ Topology information (only required ones)
-
toml_table_t*
-
typedef struct PAL_CPU_INFO_
PAL_CPU_INFO
¶
-
struct
PAL_CPU_INFO_
¶
-
typedef struct PAL_TOPO_INFO_
PAL_TOPO_INFO
¶
-
struct
PAL_TOPO_INFO_
¶
-
typedef struct PAL_MEM_INFO_
PAL_MEM_INFO
¶
-
struct
PAL_MEM_INFO_
¶
Pal APIs¶
The PAL APIs contain a number of functions that can be called from the library OS.
Memory allocation¶
The ABI includes three calls to allocate, free, and modify the permission bits on page-base virtual memory. Permissions include read, write, execute, and guard. Memory regions can be unallocated, reserved, or backed by committed memory.
-
int
DkVirtualMemoryAlloc
(PAL_PTR * addr, PAL_NUM size, PAL_FLG alloc_type, PAL_FLG prot)¶ Allocate virtual memory for the library OS and zero it out.
- Parameters
addr
:*addr
can be any valid address aligned at the allocation alignment orNULL
, in which case a suitable address will be picked automatically. Any memory previously allocated at the same address will be discarded (only if*addr
was provided). Overwriting any part of PAL memory is forbidden. On successful return*addr
will contain the allocated address (which can differ only in theNULL
case).size
: must be a positive number, aligned at the allocation alignment.alloc_type
: can be a combination of any of the PAL_ALLOC flagsprot
: can be a combination of the PAL_PROT flags
-
int
DkVirtualMemoryFree
(PAL_PTR addr, PAL_NUM size)¶ This API deallocates a previously allocated memory mapping.
Both
addr
andsize
must be non-zero and aligned at the allocation alignment.- Parameters
addr
: the addresssize
: the size
-
PAL_ALLOC
¶ Memory Allocation Flags
Values:
-
0x1
¶ Only reserve the memory
-
0x2
¶ Allocate for PAL (valid only if #IN_PAL)
-
0x3
¶
-
-
int
DkVirtualMemoryProtect
(PAL_PTR addr, PAL_NUM size, PAL_FLG prot)¶ Modify the permissions of a previously allocated memory mapping.
Both
addr
andsize
must be non-zero and aligned at the allocation alignment.- Parameters
addr
: the addresssize
: the sizeprot
: see DkVirtualMemoryAlloc
Process creation¶
The ABI includes one call to create a child process and one call to terminate the running process. A child process does not inherit any objects or memory from its parent process and the parent process may not modify the execution of its children. A parent can wait for a child to exit using its handle. Parent and child may communicate through I/O streams provided by the parent to the child at creation.
-
int
DkProcessCreate
(PAL_STR * args, PAL_HANDLE * handle)¶ Create a new process.
Loads and executes the same binary as currently executed one (
loader.preload
in case of LibOS, orpal.entrypoint
in PAL regression tests), and passes the new arguments.- Parameters
args
: an array of strings the arguments to be passed to the new process.handle
: on success contains the process handle.
TODO:
args
is only used by PAL regression tests, and should be removed at some point.
Stream creation/connect/open¶
The stream ABI includes nine calls to open, read, write, map, unmap,
truncate, flush, delete and wait for I/O streams and three calls to
access metadata about an I/O stream. The ABI purposefully does not
provide an ioctl call. Supported URI schemes include:
file:
,
pipe:
,
http:
,
https:
,
tcp:
,
udp:
,
pipe.srv:
,
http.srv
,
tcp.srv:
and
udp.srv:
.
The latter four schemes are used to open inbound I/O streams for server
applications.
-
int
DkStreamOpen
(PAL_STR uri, PAL_FLG access, PAL_FLG share_flags, PAL_FLG create, PAL_FLG options, PAL_HANDLE * handle)¶ Open/create a stream resource specified by
uri
Supported URI types:
file:...
,dir:...
: Files or directories on the host file system. If PAL_CREATE_TRY is given increate
flags, the file/directory will be created.dev:...
: Open a device as a stream. For example,dev:tty
represents the standard I/O.pipe.srv:<name>
,pipe:<name>
,pipe:
: Open a byte stream that can be used for RPC between processes. The server side of a pipe can accept any number of connections. Ifpipe:
is given as the URI (i.e., without a name), it will open an anonymous bidirectional pipe.tcp.srv:<ADDR>:<PORT>
,tcp:<ADDR>:<PORT>
: Open a TCP socket to listen or connect to a remote TCP socket.udp.srv:<ADDR>:<PORT>
,udp:<ADDR>:<PORT>
: Open a UDP socket to listen or connect to a remote UDP socket.
- Return
- 0 on success, negative error code on failure.
- Parameters
uri
: is the URI of the stream to be opened/createdaccess
: can be a combination of the PAL_ACCESS flagsshare_flags
: can be a combination of the PAL_SHARE flagscreate
: can be a combination of the PAL_CREATE flagsoptions
: can be a combination of the PAL_OPTION flagshandle[out]
: if the resource is successfully opened or created, a PAL handle is returned in*handle
for further access such as reading or writing.
-
int
DkStreamWaitForClient
(PAL_HANDLE handle, PAL_HANDLE * client)¶ Blocks until a new connection is accepted and returns the PAL handle for the connection.
This API is only available for handles that are opened with
pipe.srv:...
,tcp.srv:...
, andudp.srv:...
.- Parameters
handle
: handle to accept a new connection on.client
: on success holds handle for the new connection.
-
int
DkStreamRead
(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM * count, PAL_PTR buffer, PAL_PTR source, PAL_NUM size)¶ Read data from an open stream.
If
handle
is a directory, DkStreamRead fills the buffer with the null-terminated names of the directory entries.- Return
- 0 on success, negative error code on failure.
- Parameters
handle
: handle to the stream.offset
: offset to read at. Ifhandle
is a file,offset
must be specified at each call.count
: on function call should contain the size ofbuffer
. On successful return contains the number of bytes read.buffer
: pointer to the buffer to read into.source
: ifhandle
is a UDP socket,size
is not zero andsource
is not NULL, the remote socket address is returned in it.size
: size of thesource
buffer.
-
int
DkStreamWrite
(PAL_HANDLE handle, PAL_NUM offset, PAL_NUM * count, PAL_PTR buffer, PAL_STR dest)¶ Write data to an open stream.
- Return
- 0 on success, negative error code on failure.
- Parameters
handle
: handle to the stream.offset
: offset to write to. Ifhandle
is a file,offset
must be specified at each call.count
: on function call should contain the size ofbuffer
. On successful return contains the number of bytes written.buffer
: pointer to the buffer to write from.dest
: if the handle is a UDP socket, specifies the remote socket address.
-
int
DkStreamDelete
(PAL_HANDLE handle, PAL_FLG access)¶ Delete files or directories on the host or shut down the connection of TCP/UDP sockets.
- Parameters
access
: which side to shut down (PAL_DELETE), or both if 0 is given.
-
int
DkStreamMap
(PAL_HANDLE handle, PAL_PTR * addr, PAL_FLG prot, PAL_NUM offset, PAL_NUM size)¶ Map a file to a virtual memory address in the current process.
- Return
- 0 on success, negative error code on failure.
- Parameters
handle
: handle to the stream to be mapped.addr
: see DkVirtualMemoryAllocprot
: see DkVirtualMemoryAllocoffset
: offset in the stream to be mapped. Must be properly aligned.size
: size of the requested mapping. Must be non-zero and properly aligned.
-
int
DkStreamUnmap
(PAL_PTR addr, PAL_NUM size)¶ Unmap virtual memory that is backed by a file stream.
addr
andsize
must be aligned at the allocation alignment- Return
- 0 on success, negative error code on failure.
-
int
DkStreamSetLength
(PAL_HANDLE handle, PAL_NUM length)¶ Set the length of the file referenced by handle to
length
.- Return
- 0 on success, negative error code on failure.
-
int
DkStreamFlush
(PAL_HANDLE handle)¶ Flush the buffer of a file stream.
- Return
- 0 on success, negative error code on failure.
-
int
DkSendHandle
(PAL_HANDLE handle, PAL_HANDLE cargo)¶ Send a PAL handle over another handle.
Currently, the handle that is used to send cargo must be a process handle.
- Return
- 0 on success, negative error code on failure.
- Parameters
cargo
: the handle being sent
-
int
DkReceiveHandle
(PAL_HANDLE handle, PAL_HANDLE * cargo)¶ This API receives a handle over another handle.
TODO: document usage and parameters.
- Return
- 0 on success, negative error code on failure.
-
int
DkStreamAttributesQuery
(PAL_STR uri, PAL_STREAM_ATTR * attr)¶ Query the attributes of a named stream.
This API only applies for URIs such as
file:...
,dir:...
, anddev:...
.
-
typedef struct _PAL_STREAM_ATTR
PAL_STREAM_ATTR
¶
-
struct
_PAL_STREAM_ATTR
¶
-
int
DkStreamAttributesQueryByHandle
(PAL_HANDLE handle, PAL_STREAM_ATTR * attr)¶ Query the attributes of an open stream.
This API applies to any stream handle.
-
int
DkStreamAttributesSetByHandle
(PAL_HANDLE handle, PAL_STREAM_ATTR * attr)¶ Set the attributes of an open stream.
-
int
DkStreamGetName
(PAL_HANDLE handle, PAL_PTR buffer, PAL_NUM size)¶ Query the name of an open stream. On success
buffer
contains a null-terminated string.
-
int
DkStreamChangeName
(PAL_HANDLE handle, PAL_STR uri)¶ This API changes the name of an open stream.
Flags used for stream manipulation¶
-
PAL_ACCESS
¶ Stream Access Flags
Values:
-
PAL_ACCESS_RDONLY
¶
-
PAL_ACCESS_WRONLY
¶
-
PAL_ACCESS_RDWR
¶
-
PAL_ACCESS_BOUND
¶
-
-
PAL_SHARE
¶ Stream Sharing Flags
Values:
-
01
¶
-
02
¶
-
04
¶
-
010
¶
-
020
¶
-
040
¶
-
0100
¶
-
0200
¶
-
0400
¶
-
01000
¶
-
02000
¶
-
04000
¶
-
07777
¶
-
-
PAL_CREATE
¶ Stream Create Flags
Values:
-
1
¶ Create file if file does not exist
-
2
¶ Create file and fail if file already exists
-
4
¶ Create dual-stack socket (opposite of IPV6_V6ONLY)
-
7
¶
-
-
PAL_OPTION
¶ Stream Option Flags
Values:
-
1
-
2
specific to
eventfd
syscall
-
4
-
7
-
-
PAL_DELETE
¶ Values:
-
1
shut down the read side only
-
2
shut down the write side only
-
Thread creation¶
The ABI supports multithreading through five calls to create, sleep, yield the scheduler quantum for, resume execution of, and terminate threads, as well as seven calls to create, signal, and block on synchronization objects.
-
int
DkThreadCreate
(PAL_PTR addr, PAL_PTR param, PAL_HANDLE * handle)¶ Create a thread in the current process.
- Parameters
addr
: is the address of an entry point of execution for the new threadparam
: is the pointer argument that is passed to the new threadhandle
: on success contains the thread handle
-
void
DkThreadYieldExecution
(void)¶ Yield the current thread such that the host scheduler can reschedule it.
-
void
DkThreadExit
(PAL_PTR clear_child_tid)¶ Terminate the current thread.
- Parameters
clear_child_tid
: is the pointer to memory that is erased on thread exit to notify LibOS (which in turn notifies the parent thread if any); ifclear_child_tid
is NULL, then PAL doesn’t do the clearing.
-
int
DkThreadResume
(PAL_HANDLE thread)¶ Resume a thread.
Exception handling¶
-
PAL_EVENT
¶ Values:
-
1
arithmetic error (div-by-zero, floating point exception, etc.)
-
PAL_EVENT_MEMFAULT
¶ segmentation fault, protection fault, bus fault
-
PAL_EVENT_ILLEGAL
¶ illegal instructions
-
PAL_EVENT_QUIT
¶ terminated by external program (see “sys.enable_sigterm_injection” manifest option)
-
PAL_EVENT_INTERRUPTED
¶ interrupted (usually internally to handle aync event)
-
PAL_EVENT_NUM_BOUND
¶
-
-
typedef struct PAL_CONTEXT
PAL_CONTEXT
¶
-
struct
PAL_CONTEXT
-
typedef void
(* PAL_EVENT_HANDLER)
(bool is_in_pal, PAL_NUM addr, PAL_CONTEXT *context)¶ Type of exception handlers (upcalls).
- Parameters
is_in_pal
:true
if the exception happened inside PALaddr
: address of the exception (meaningful only for sync exceptions)context
: CPU context at the moment of exception.
-
void
DkSetExceptionHandler
(PAL_EVENT_HANDLER handler, PAL_NUM event)¶ Set the handler for the specific exception event.
- Parameters
event
: can be one of PAL_EVENT values
Synchronization¶
-
NO_TIMEOUT
¶ block until the handle’s event is triggered
-
int
DkEventCreate
(PAL_HANDLE * handle, bool init_signaled, bool auto_clear)¶ Create an event handle.
Creates a handle to an event that resembles WinAPI synchronization events. A thread can set (signal) the event using
DkEventSet, clear (unset) it using DkEventClear or wait until the event becomes set (signaled) using DkEventWait.- Parameters
handle
: on success*handle
contains pointer to the event handleinit_signaled
: initial state of the event (true
- set,false
- not set)auto_clear
:true
if a successful wait for the event should also reset (consume) it
-
void
DkEventSet
(PAL_HANDLE handle)¶ Set (signal) an event.
If the event is already set, does nothing.
This function has release semantics and synchronizes with DkEventWait.
-
void
DkEventClear
(PAL_HANDLE handle)¶ Clear (unset) an event.
If the event is not set, does nothing.
-
int
DkEventWait
(PAL_HANDLE handle, uint64_t * timeout_us)¶ Wait for an event handle.
timeout_us
points to a value that specifies the maximal time (in microseconds) that this function should sleep if this event is not signaled in the meantime. SpecifyingNULL
blocks indefinitely. Note that in any case this function can return earlier, e.g. if a signal has arrived, but this will be indicated by the returned error code. After returning (both successful and not),timeout_us
will contain the remaining time (time that need to pass before we hit originaltimeout_us
).- Return
- 0 if the event was triggered, negative error code otherwise (#PAL_ERROR_TRYAGAIN in case of timeout triggering)
- Parameters
handle
: handle to wait on, must be of “event” typetimeout_us
: timeout for the wait
This function has acquire semantics and synchronizes with DkEventSet.
Objects¶
-
int
DkStreamsWaitEvents
(PAL_NUM count, PAL_HANDLE * handle_array, PAL_FLG * events, PAL_FLG * ret_events, PAL_NUM timeout_us)¶ Poll.
- Return
- 0 if there was an event on at least one handle, negative error code otherwise
- Parameters
count
: the number of items in the arrayhandle_array
:events
: user-defined eventsret_events
: polled-handles’ events inret_events
timeout_us
: is the maximum time that the API should wait (in microseconds), orNO_TIMEOUT
to indicate it is to be blocked until at least one handle is ready.
-
void
DkObjectClose
(PAL_HANDLE objectHandle)¶ Close (deallocate) a PAL handle.
Miscellaneous¶
The ABI includes seven assorted calls to get wall clock time, generate cryptographically-strong random bits, flush portions of instruction caches, increment and decrement the reference counts on objects shared between threads, and to obtain an attestation report and quote.
-
int
DkDebugLog
(PAL_PTR buffer, PAL_NUM size)¶ Output a message to the debug stream.
- Return
- 0 on success, negative error code on failure.
- Parameters
buffer
: message to write.size
:buffer
size.
-
const PAL_CONTROL*
DkGetPalControl
(void)¶
-
int
DkSystemTimeQuery
(PAL_NUM * time)¶ Get the current time.
- Parameters
time
: on success holds the current time in microseconds
-
int
DkRandomBitsRead
(PAL_PTR buffer, PAL_NUM size)¶ Cryptographically secure random.
- Return
- 0 on success, negative on failure
- Parameters
buffer
: is filled with cryptographically-secure random valuessize
: buffer size
-
int
DkSegmentRegisterGet
(PAL_FLG reg, PAL_PTR * addr)¶ Get segment register.
- Return
- 0 on success, negative error value on failure
- Parameters
reg
: the register to get (#PAL_SEGMENT)addr
: the address where result will be stored
-
int
DkSegmentRegisterSet
(PAL_FLG reg, PAL_PTR addr)¶ Set segment register.
- Return
- 0 on success, negative error value on failure
- Parameters
reg
: the register to be set (#PAL_SEGMENT)addr
: the address to be set
-
PAL_NUM
DkMemoryAvailableQuota
(void)¶ Return the amount of currently available memory for LibOS/application usage.
-
int
DkCpuIdRetrieve
(PAL_IDX leaf, PAL_IDX subleaf, PAL_IDX values[CPUID_WORD_NUM])¶ Return CPUID information, based on the leaf/subleaf.
- Parameters
values
: the array of the results
-
int
DkAttestationReport
(PAL_PTR user_report_data, PAL_NUM * user_report_data_size, PAL_PTR target_info, PAL_NUM * target_info_size, PAL_PTR report, PAL_NUM * report_size)¶ Obtain the attestation report (local) with
user_report_data
embedded into it.Currently works only for Linux-SGX PAL, where
user_report_data
is a blob of exactly 64B,target_info
is an SGX target_info struct of exactly 512B, andreport
is an SGX report obtained via the EREPORT instruction (exactly 432B). Iftarget_info
contains all zeros, then this function additionally returns this enclave’s target info intarget_info
. Useful for local attestation.The caller may specify
*user_report_data_size
,*target_info_size
, and*report_size
as 0 and other fields as NULL to get PAL-enforced sizes of these three structs.- Parameters
user_report_data
: Report data with arbitrary contents (typically uniquely identifies this Gramine instance). Must be a 64B buffer in case of SGX PAL.user_report_data_size
: Caller specifies size ofuser_report_data
; on return, contains PAL-enforced size ofuser_report_data
(64B in case of SGX PAL).target_info
: Target info of target enclave for attestation. If it contains all zeros, it is populated with this enclave’s target info. Must be a 512B buffer in case of SGX PAL.target_info_size
: Caller specifies size oftarget_info
; on return, contains PAL-enforced size oftarget_info
(512B in case of SGX PAL).report
: Attestation report withuser_report_data
embedded, targeted for an enclave with providedtarget_info
. Must be a 432B buffer in case of SGX PAL.report_size
: Caller specifies size ofreport
; on return, contains PAL-enforced size ofreport
(432B in case of SGX PAL).
-
int
DkAttestationQuote
(PAL_PTR user_report_data, PAL_NUM user_report_data_size, PAL_PTR quote, PAL_NUM * quote_size)¶ Obtain the attestation quote with
user_report_data
embedded into it.Currently works only for Linux-SGX PAL, where
user_report_data
is a blob of exactly 64B andquote
is an SGX quote obtained from Quoting Enclave via AESM service.- Parameters
user_report_data
: Report data with arbitrary contents (typically uniquely identifies this Gramine instance). Must be a 64B buffer in case of SGX PAL.user_report_data_size
: Size in bytes ofuser_report_data
. Must be exactly 64B in case of SGX PAL.quote
: Attestation quote withuser_report_data
embedded.quote_size
: Caller specifies maximum size allocated forquote
; on return, contains actual size of obtained quote.
-
int
DkSetProtectedFilesKey
(PAL_PTR pf_key_hex)¶ Set wrap key (master key) for protected files.
Currently works only for Linux-SGX PAL. This function is supposed to be called during remote attestation and secret provisioning, before the user application starts.
- Parameters
pf_key_hex
: Wrap key for protected files. Must be a 32-char null-terminated hex string in case of SGX PAL (AES-GCM encryption key).