2. Configuring the Server¶
Most configuration parameters have reasonable defaults that work in many cases, and installations generally won’t need to change them. The main exceptions to this guidance are the security information:
The defined security labels and “group-label” connections may need to be updated to reflect a site’s specific user base and user-authentication system. Details of this set up are in the Configuring Groups and Labels section.
The AWS credential information has no reasonable defaults for securely accessing data in S3 buckets.
To run with SSL encryption between the Python client and the xGT server the configuration parameters for
system.hostname,system.ssl_root_dir, andsystem.usesslmust be provided.
Configuring the xGT server is done using key-value pairs.
The pairs can be given either in configuration files written in JSON format or as command line arguments when launching xgtd, the xGT server process, by modifying /etc/systemd/system/xgtd.service.
The standard configuration file is located at /etc/xgtd/xgtd.conf.
Administrators may split configuration information into multiple files, which need to be passed to the xGT server as program arguments, for example: -c /path/to/xgtd.part1.conf.
It is also possible to specify individual key-value pairs in the form of a program argument: -Dkey=value or -Dkey:value.
The -c and -D program arguments can appear multiple times.
The algorithm that xGT uses to establish the working configuration is to process the -c and -D program arguments in the left-to-right order they appear on the ExecStart statement in the /etc/systemd/system/xgtd.service file.
The default is to have only the -c /etc/xgtd/xgtd.conf configuration argument.
For any key that appears multiple times (from any combination of configuration files or as -D program arguments), the last configuration value for that key is used.
That is, repeated key entries will overwrite previous entries in the working configuration.
An example ExecStart statement is:
ExecStart=/usr/bin/numactl --interleave=all -- /opt/xgtd/bin/xgtd -c /etc/xgtd/xgtd.conf -c /etc/xgtd/logging.conf -Dlogging.level.transactions=debug
The following keys for configuring xGT are supported. The “Boot Only” column indicates if the configuration variable can only be set at system startup. Changes to the value of a boot only variable after system startup will be ignored.
Key |
Type |
Default |
Boot Only |
Description |
|---|---|---|---|---|
arrow.transfer_compression |
string |
zstd |
Controls compression of Arrow data transfers between the Python client and the xGT server, in both directions. The values are “zstd”, “lz4”, and “none”. Details and guidance are in Network Transfer Compression. |
|
audit.config_file |
string |
Container: /conf/audit.xml |
Y |
Provides the log4cxx configuration file name. |
aws.access_key_id |
string |
Y |
Access key id for accessing Amazon S3 datastores. |
|
aws.secret_access_key |
string |
Y |
Secret key for accessing Amazon S3 datastores. |
|
jobs.fairshare_halflife_secs |
int |
600 |
Half-life, in seconds, of the decaying resource-usage total each user accrues while their jobs run. A longer half-life makes fair-share scheduling weigh usage further into the past. 0 disables fair-share tracking. |
|
jobs.job_visibility |
string |
redacted |
Controls how much of other users’ jobs a non-administrator can see through the job-info APIs. “private” shows only the user’s own jobs. “redacted” additionally shows other users’ active jobs with status, timestamps, and priority only, without the owner or query content. “full” shows other users’ active jobs in full. Administrators always see all jobs regardless of this setting. |
|
jobs.max_executing |
int |
0 |
Maximum number of user jobs that may execute concurrently. Additional runnable jobs wait in the scheduling queue. 0 selects an automatic default of half the worker-thread count (at least 1). |
|
jobs.max_suspended |
int |
100 |
Maximum number of jobs the scheduler may hold suspended at once to make room for higher-priority or time-sliced work. 0 disables suspension; -1 means no limit. |
|
jobs.priority_age_horizon_secs |
int |
240 |
The queue wait time, in seconds, at which a job’s age factor reaches its full weight. With the default weights, a job that has waited this long outranks a fresh submission regardless of its priority. The minimum is 1. |
|
jobs.priority_weight_age |
int |
1000 |
Relative weight of a queued job’s waiting time in the scheduler’s admission score. Raising it lets long-waiting jobs advance sooner relative to higher-priority arrivals. |
|
jobs.priority_weight_fairshare |
int |
1000 |
Relative weight of the owner’s recent resource usage (fair share) in the scheduler’s admission score. Raising it more strongly favors users who have run less recently. |
|
jobs.priority_weight_nice |
int |
1000 |
Relative weight of a job’s priority in the scheduler’s admission score. Raising it makes the user-set priority dominate age and fair share. |
|
jobs.small_io_max_bytes |
int |
2147483648 |
Upper bound, in bytes, on the estimated payload for an egest job to use the small-I/O fast lane, a separate worker pool so small result downloads are not queued behind large jobs. 0 skips the payload-size check and applies the row bound only. |
|
jobs.small_io_max_rows |
int |
5000000 |
Upper bound, in rows, for an egest job to qualify for the small-I/O fast lane. 0 classifies nothing as small I/O, so all egest jobs use the main worker pool. |
|
jobs.time_slice_secs |
int |
30 |
Time-slice quantum in seconds. Once a running job has held an execution slot for this long, the scheduler may rotate it out in favor of a waiting job so no job starves. 0 disables time-slicing. |
|
license.core_count |
int |
Number of physical cores on the system |
Y |
Controls how many cores to check out from the license manager. Only works with floating licenses designated with LicenseType=cores. |
license.feature |
string |
Standard xgtd license feature |
Y |
Controls the feature to check out in a license. |
license.location |
string |
Container: /conf/licenses/xgtd.lic |
Y |
Controls the path xgtd will look for when checking out a license. The format is file location. Multiple paths may be given, but should be separated by a colon (:). If multiple paths are given, local licenses will be preferred over network licenses. |
license.socket_count |
int |
Number of physical sockets on the system |
Y |
Controls how many sockets to check out from the license manager. Only works with floating licenses designated with LicenseType=sockets. |
logging.filepath |
string |
Container: /log/ |
Y |
Provides the directory path where the log file will be written. If the filepath is “stdout” or “stderr”, then the log stream is written to standard out or standard error. |
logging.fileprefix |
string |
xgtd |
Y |
Provides the prefix of the log file name. |
logging.level.<component> |
string |
warning |
Sets the level of detail to place into a log stream for the given logging component. The components are described in Logging Components. The levels are: “debug”, “info”, “warning”, “fatal”. |
|
logging.max_file_size |
int |
10,485,760 (10 MiB) |
Y |
Sets the maximum logging file size in bytes before it is rotated. |
logging.max_files |
int |
10 |
Y |
Sets the maximum number of archived log files (older files are deleted as new ones are rotated in). |
metrics.cache |
bool |
true |
Controls whether the metrics cache is running and generating statistical information about each frame. These statistics are used for query order optimization. |
|
security.adminlabel |
string |
xgtadmin |
Y |
Value for the administrator label. |
security.allow_empty_labels |
bool |
false |
Controls whether empty CRUD labels are allowed for frames and namespaces created by regular users. |
|
security.inherit_namespace_labels |
bool |
true |
When creating a frame with no security labels inside a namespace that has labels, add those labels to the created frame. |
|
security.grouplabelfile |
string |
Container: /conf/grouplabel.csv |
Y |
The location of the group label file listing the mappings from groups to security labels to be loaded at boot time. |
security.labelfile |
string |
Container: /conf/label.csv |
Y |
The location of the label file listing the security labels to be loaded at boot time. |
security.oidc.audience |
string |
Y |
The expected audience value in the token. [1]
Must match the |
|
security.oidc.ca_cert |
string |
Y |
Path to a CA bundle file for HTTPS requests the server makes to the IdP [1] (discovery, JWKS, introspection, userinfo). Use when the IdP uses a private or self-signed certificate. See OIDC Authentication. |
|
security.oidc.client_id |
string |
Y |
The OAuth2 client ID registered with the IdP. [1]
Returned to clients via |
|
security.oidc.groups_claim |
string |
groups |
Y |
The claim name containing the user’s group memberships. [1] Used to compute security labels (see Group Membership). Dot notation is supported for nested claims. Leave empty to disable group extraction. See OIDC Authentication. |
security.oidc.introspection_client_id |
string |
Y |
Client ID used to authenticate against the introspection endpoint. [1] Required by introspection-based validation modes when the IdP requires client authentication. See OIDC Authentication. |
|
security.oidc.introspection_client_secret |
string |
Y |
Client secret used to authenticate against the introspection endpoint. [1] Required by introspection-based validation modes when the IdP requires client authentication. See OIDC Authentication. |
|
security.oidc.introspection_uri |
string |
Y |
Override the token introspection endpoint URL. [1] If not set, taken from the IdP discovery document. Required for introspection-based modes if discovery is unavailable. See OIDC Authentication. |
|
security.oidc.issuer |
string |
Y |
The OIDC issuer URL. [1]
Used to validate the |
|
security.oidc.jwks_uri |
string |
Y |
Override the JWKS endpoint URL. [1]
If not set, taken from the IdP discovery document.
Only used by the |
|
security.oidc.openshift_user_api_uri |
string |
Y |
The OpenShift User API URI used by the |
|
security.oidc.scopes |
string |
openid profile email |
Y |
Space-separated list of OAuth2 scopes advertised to clients via |
security.oidc.userinfo_uri |
string |
Y |
Override the userinfo endpoint URL. [1]
If not set, taken from the IdP discovery document.
Only used by the |
|
security.oidc.username_claim |
string |
preferred_username |
Y |
The claim name used as the xGT username. [1]
Typically |
security.oidc.validation_mode |
string |
jwt |
Y |
How the server validates incoming bearer tokens. [1]
One of |
security.server_krb_principal |
string |
Y |
Value for the Kerberos principal to use for the xgtd server. |
|
security.session_refresh_window |
int |
-1 |
The time in seconds for how often to refresh the life extent of a session. A value of -1 indicates an unlimited time meaning a refresh will always be done. |
|
security.session_ttl |
int |
-1 |
The time-to-live for the server. The maximum amount of time in seconds that a session will stay alive. A value of -1 indicates no limit. |
|
security.use_kerberos |
bool |
false |
Y |
Set to true to use Kerberos authentication for xgtd. |
system.auto_create_user_namespace |
bool |
true |
Y |
Automatically create a private namespace for each user when logging in. |
system.grpc_service_threads |
int |
16 |
Y |
Number of threads dedicated to serving latency-sensitive control requests such as job status, listing, cancellation, and submission. Keeping these separate from query execution lets those calls stay responsive while heavy jobs run. |
system.health_port |
int |
4366 |
Y |
Port for the server’s health-check endpoint, which container orchestrators and load balancers probe to determine whether the server is running. Valid values are [0, 65535]. Set to -1 to disable the health-check endpoint. |
system.hostname |
string |
localhost |
Y |
Controls the hostname (or IP address) the server will listen on for client connections. An IP address or “localhost” are allowed. |
system.io_directory |
string |
Container: /data/ |
Y |
All input/output (ingest/egest) must be done in this file system location or below (sandbox directory). |
system.io_follow_symbolic_links |
bool |
false |
Y |
Controls whether to follow symbolic links in the underlying file system. |
system.io_threads |
int |
16 |
Controls the number of I/O threads xGT uses per CSV file. The number will never exceed the system cores count including hyperthreads. |
|
system.locale |
string |
C |
Y |
Locale for comparing strings. Details provided in Configuring the Locale. |
system.max_memory |
int |
Available system RAM - min(10% system RAM, 8GiB) |
Y |
Controls the maximum amount of memory (in GiB) that xGT will use. Using more than the default is allowed but may cause system instability. The value is capped at the available system RAM. |
system.parquet_io_threads |
int |
system.worker_threads |
Controls the number of I/O threads xGT uses per Parquet file. The number will never exceed the system cores count including hyperthreads. |
|
system.parquet_max_files |
int |
1 |
Controls the number of Parquet files xGT reads simultaneously. |
|
system.port |
int |
4367 |
Y |
Controls the port the server will listen on for client connections. Valid values are [0, 65535]. |
system.small_io_worker_threads |
int |
2 |
Y |
Number of worker threads reserved for small, bounded result downloads so they are not queued behind large egest jobs. 0 disables the reserved threads and routes those downloads through the regular worker threads. |
system.ssl_root_dir |
string |
Container: /conf/ssl/ |
Y |
The path to the root directory that holds the server’s SSL certificates and private keys. |
system.table_cleanup_interval |
string |
PT0S (disabled) |
Controls how long an automatically-managed results table frame may remain idle before the server removes it.
Results tables are those created to hold query results; frames created explicitly (for example, via |
|
system.usemtls |
bool |
false |
Y |
Controls whether to use stricter mutual TLS authentication between the client and server. |
system.usessl |
bool |
false |
Y |
Controls whether to turn on SSL authentication between the client and server. |
system.worker_threads |
int |
Number of cores including hyperthreads on the system or the maximum allowed by the license. |
Y |
Controls the number of worker threads xGT uses. The number will never exceed the system cores count including hyperthreads. |
system.pin_threads |
bool |
false |
Y |
Internal use only. |
The log filename is created using both logging.filepath and logging.fileprefix.
Assuming a file prefix of “prefix” and a file path of “/path/to”, the log file will be created with a name: /path/to/prefix.log.
The logger will periodically rotate by renaming and then compressing the renamed log file, along with opening a new file with the original name.
The maximum size of each log file is configured using the parameter logging.max_file_size.
The maximum number of rotated log files is configured using the parameter logging.max_files.
If this maximum is reached, older rotated files are deleted.
Consider this example configuration file located at /path/to/xgtd.conf:
{
"system": {
"worker_threads": 16,
"io_threads": 10,
"port": 4369
},
"system.locale": "en_US.UTF-8"
}
Launching the xgtd daemon is done by invoking this command:
$ systemctl start xgtd
and is controlled by the /etc/systemd/system/xgtd.service file.
To change the configuration of xgtd, either modify the standard configuration file at /etc/xgtd/xgtd.conf or update the ExecStart command in the systemctl service file.
To launch xgtd using the example configuration file shown above and supplementing with a definition for system.hostname, modify etc/systemd/system/xgtd.service as follows:
$ ExecStart=/opt/xgtd/bin/xgtd -c /path/to/xgtd.conf -Dsystem.hostname=127.0.0.1
To launch xgtd using this configuration file but overriding the worker_threads configuration value:
$ ExecStart=/opt/xgtd/bin/xgtd -c /path/to/xgtd.conf -Dsystem.worker_threads=8
To launch xgtd using multiple configuration files:
$ ExecStart=/opt/xgtd/bin/xgtd -c /path/to/xgtd_conf1.conf -Dsystem.port=4369 -c /path/to/xgtd_conf2.conf
2.1. Configuring Job Scheduling¶
The xGT server runs submitted jobs through a scheduler that decides which queued jobs start, how many run at once, and how long-running jobs share the machine.
Every scheduling parameter has a reasonable default (listed in the table above), and most sites never need to change them.
All of the jobs.* parameters can be changed while the server is running, so tuning them does not require a restart.
Execution capacity.
jobs.max_executing bounds how many jobs execute concurrently; additional runnable jobs wait in the scheduling queue.
A value of 0 (the default) selects half the worker-thread count.
When every slot is in use, the scheduler makes room for higher-priority or longer-waiting work by temporarily suspending running jobs, up to jobs.max_suspended at a time; a value of 0 disables suspension entirely.
Time-slicing.
So that no single job monopolizes the machine, a job that has held an execution slot for jobs.time_slice_secs may be rotated out in favor of a waiting job and resumed later.
Setting it to 0 disables time-slicing, so a running job keeps its slot until it finishes.
Small result downloads.
Fetching a small query result should not have to wait behind large data-movement jobs, so an egest job whose output is under both jobs.small_io_max_rows and jobs.small_io_max_bytes runs in a separate fast lane backed by its own worker threads, system.small_io_worker_threads.
Setting the worker count to 0 disables the lane and routes those downloads through the regular workers.
The thresholds bound what qualifies: jobs.small_io_max_rows of 0 classifies nothing as small, and jobs.small_io_max_bytes of 0 applies the row bound only.
Priority. Each job carries a priority from 1 to 40, where higher values are scheduled sooner and the default is 20. Clients set a job’s priority when submitting it and may change it afterward. A user may set any priority in the range 1 to 20 on their own jobs to reorder their own work; raising a job above 20, or changing another user’s job, requires administrator privilege.
Fairness.
Priority is not the only input to scheduling.
To keep things fair, the scheduler also favors jobs that have waited a long time and users who have run little recently, so a low-priority job is never stuck forever and no single user can monopolize the machine.
This is on by default and suits most sites.
Administrators who need to tune it can change how strongly each input counts: the weights jobs.priority_weight_nice for priority, jobs.priority_weight_age for wait time, and jobs.priority_weight_fairshare for recent usage.
Two more parameters control the timing: jobs.priority_age_horizon_secs sets how fast a waiting job gains ground, and jobs.fairshare_halflife_secs sets how fast usage history fades.
The table above lists the defaults and, where applicable, valid ranges.
Responsiveness of job control.
Submitting jobs and querying, listing, or canceling them are served on a dedicated thread pool, system.grpc_service_threads, kept separate from job execution so these control operations stay responsive even while the machine is busy running jobs.
Visibility of other users’ jobs.
jobs.job_visibility controls how much of another user’s jobs a non-administrator can see through the job-information APIs.
private shows only the caller’s own jobs; redacted (the default) additionally shows other users’ active jobs with status, timestamps, and scheduling priority only, without the owner or query content; and full shows those active jobs in full.
Administrators always see all jobs regardless of this setting.
2.2. Configuring Groups and Labels¶
The collection of groups and labels is used extensively by the User Authentication system. Users are not configured in xGT, relying instead on some external authentication system such as LDAP to manage the users. This external authentication system is queried for a list of groups of which an authenticated user is a member. The xGT application maintains recognized groups, security labels (one may imagine these to be names of roles), and which groups possess which labels.
These labels and group-to-label relationships are held in CSV files that are read into xGT when it launches.
The configuration variables security.labelfile and security.grouplabelfile indicate where xGT should find these CSV files.
To establish appropriate security configuration, the content of the two CSV files should be updated.
It is recommended that these CSV files contain the “master” copy of the groups and labels information.
It is possible to alter the online data frames holding this information (with the appropriate administrator privilege), but the only way this information is remembered across application restarts is if the information is stored into the CSV files.
An authenticated user may belong to several groups, each of which is associated with many security labels. The union of the labels that are reachable through the groups that the user belongs to is called the user’s label set.
An example group and label setup is shown below. The groups listed are those that exist in an external authentication system such as LDAP. In this example, a user that belongs to group1 and group3 has a label set consisting of labels labelA, labelB, and labelD. A user that belongs to group adminGroup has the label xgtadmin in their label set.
Example contents of security.labelfile:
xgtadmin
labelA
labelB
labelC
labelD
Example contents of security.grouplabelfile:
adminGroup, xgtadmin
group1, labelA
group1, labelB
group2, labelC
group2, labelD
group3, labelD
2.3. Configuring Administrator Privileges¶
Within the xGT application, there is a special security label that gives administrator privilege.
This security label can be set by the security.adminlabel configuration variable, and its default is “xgtadmin”.
Any user that has the configured security.adminlabel in their label set has administrator privileges.
By default, xGT is configured to authenticate users using their local UNIX credentials.
Section User Authentication explains in more detail how authentication operates for the xGT server.
To configure an xGT site administrator, the group xgtgroup can be created locally on the Linux system running xGT and added to any existing UNIX user on that system.
xgtgroup is the default name of the group that maps to the administrator label xgtadmin.
A Linux user belonging to that group can now log in to xGT using their regular Linux credentials and will have system administrator privileges inside the xGT server.
The default contents of the security.labelfile provided with the xGT installation are as follows:
xgtadmin
The default contents of the security.grouplabelfile provided with the xGT installation are as follows:
xgtgroup,xgtadmin
Any user that has the security.adminlabel in their label set has unrestricted visibility and access to any data in the server.
Any access control check with a user that holds the security.adminlabel behaves as though the user holds all possible labels.
2.4. Configuring the Locale¶
The system.locale (locale) is used by xGT to impact how to compare two different strings according to the lexicographic comparison rules defined by the locale.
The default value for locale is “C”, which essentially means using the standard C library for string comparisons.
The xGT configuration mechanism will check every configured value for semantic correctness. This means that before a configured value is accepted as valid, xGT will try to configure the underlying C++ and xGT runtime systems passing in the configured value. If no exception is thrown, the string is accepted as a valid locale value.
2.5. Using an SSL Secure Channel¶
By default, xGT uses an insecure channel to communicate between the client and server, but an admin or user can enable a secure channel using SSL certificates.
2.5.1. Server Setup¶
To run a secure server, set the following config variable in the xgtd configuration file (e.g. xgtd.conf):
"system.usessl" : true,
The server’s SSL certificates and private key should be placed in the root SSL directory. The xGT server expects the following directory structure in the root SSL directory:
.
├── certs
│ ├── ca-chain.cert.pem # Certificate chain.
│ └── server.cert.pem # Server public certificate.
└── private
└── server.key.pem # Server private key.
The container version of xGT uses /conf/ssl as the root SSL directory, while the RPM version uses /etc/xgtd/ssl.
If needed, the default location can be changed by setting the configuration variable:
.. code-block:
"system.ssl_root_dir" : "/path/to/server/certs/root",
2.5.2. Client Setup¶
To connect to an xGT server using SSL, the client needs to pass the following values in the flags parameter to the Connection constructor:
The
sslflag needs to be set toTrue.Either the
ssl_root_dirflag should be set to the root directory containing the SSL certificates and private keys, or thessl_server_certflag should be set to the path to the certificate chain that validates the server’s certificate.The
ssl_server_cnflag can be set to the common name for the server listed on the server-side SSL certificate but is usually only necessary for self-signed certificates.
Here’s an example using the BasicAuth authentication type and a certificate signed by a trusted Certificate Authority:
import xgt
server = xgt.Connection(
auth = xgt.BasicAuth('username', 'password'),
flags = { 'ssl' : True,
'ssl_root_dir' : "/path/to/client/certs/root" })
Here’s an example using a self-signed certificate and providing the path to the certificate chain instead of the key directory:
import xgt
server = xgt.Connection(
auth = xgt.BasicAuth('username', 'password'),
flags = { 'ssl' : True,
'ssl_server_cert' : "/path/to/client/certificate_chain",
'ssl_server_cn' : "MyCommonName" })
The xGT client expects the following directory structure for SSL certificates and private keys, when providing a directory location with the ssl_root_dir flag:
.
├── certs
│ ├── ca-chain.cert.pem # Certificate chain.
│ └── client.cert.pem # Client public certificate for mTLS.
└── private
└── client.key.pem # Client private key for mTLS.
Note that the client public certificate and private key are only needed when using mutual TLS (mTLS).
2.5.3. Mutual TLS¶
For added security, xGT can be set up to require mutual TLS which requires the client to also have a public certificate and a private key to verify their identity. Enabling mutual TLS requires changes on both the server and client.
For the server, set the following config variable in the xgtd configuration file (e.g. xgtd.conf):
"system.usemtls" : true,
For the client if the ssl_root_dir flag is used, set the flag ssl_use_mtls to True. Here’s an example:
import xgt
server = xgt.Connection(
auth = xgt.BasicAuth('username', 'password'),
flags = { 'ssl' : True,
'ssl_root_dir' : "/path/to/client/certs/root",
'ssl_use_mtls' : True })
If passing certificate and key locations, the ssl_client_cert and ssl_client_key flags must also be passed given values of the paths to the client public certificate and private key, respectively. Here’s an example:
import xgt
server = xgt.Connection(
auth = xgt.BasicAuth('username', 'password'),
flags = { 'ssl' : True,
'ssl_server_cert' : "/path/to/client/certificate_chain",
'ssl_use_mtls' : True,
'ssl_client_cert' : "/path/to/client/client.cert.pem",
'ssl_client_key' : "/path/to/client/client.key.pem" })
2.5.4. Using PKI Authentication¶
When both an SSL connection and mTLS are enabled by the server, the PKIAuth authentication type can be used.
Using PKIAuth requires the userId field in the certificate to be set which is how the user’s identity is determined.
Here’s an example passing the certificate / key root directory using a certificate signed by a trusted Certificate Authority:
import xgt
server = xgt.Connection(
auth = xgt.PKIAuth(ssl_root_dir = "/path/to/client/certs/root"))
Here’s an example passing the certificate / key locations using a self-signed certificate:
import xgt
server = xgt.Connection(
auth = xgt.PKIAuth(
ssl_server_cert = "/path/to/client/certificate_chain",
ssl_client_cert = "/path/to/client/client.cert.pem",
ssl_client_key = "/path/to/client/client.key.pem"),
flags = { 'ssl_server_cn' : "MyCommonName" })
Footnotes