2.2. Frames, Namespaces and Graphs

xGT uses the concept of a frame as a fundamental building block for its data storage. A frame is a structured collection of homogeneous data in which all elements in the frame, called rows, have the same number of elements. A row in a frame has a number of columns, each of which can have one of the fundamental xGT data types (Data Types and Operators).

Frames in xGT have several properties associated with them: a schema, which consists of a description of the frame’s columns, their names and data types, and a name. This means that each vertex or edge in a single frame has the same set of properties.

Frame names in xGT are represented using a two-level naming scheme. The first part of the name represents the namespace in which the frame is stored, while the second part of the name represents the frame’s name proper. Frame names need to be unique only within their own namespace.

xGT uses the term “fully qualified name” to indicate the fully specified name of a frame, including its namespace. Two underscores “__” are used to separate the namespace of a frame from its name in a fully qualified name: namespace1__FrameA, namespace2__FrameA. If a frame is within the default namespace, the namespace can be omitted. For example, if the default namespace is namespace1, then FrameA can be used in the place of namespace1__FrameA. Throughout most of the documentation, examples will assume that frames are in the default namespace. For more information on the default namespace, see Default Namespace.

2.2.1. Types of Frames

xGT supports three fundamental types of frames to store user data in the system:

• TableFrame: Tables represent a collection of rows with the only restriction being that they conform to the table’s schema.

• VertexFrame: Vertex frames represent a collection of rows conforming to a schema, with the additional restriction that the values of the key column must be unique across the entire vertex frame. Each row in a vertex frame corresponds to a vertex in xGT’s graph model.

• EdgeFrame: Edge frames also represent a collection of rows conforming to a schema, with the restrictions that the values of two columns in each row must map to the values of the key columns in the source and target vertex frame, respectively. Multiple rows in an edge frame can have the same values for source and target key columns. Each row in an edge frame corresponds to an edge in xGT’s graph model.

Each frame can be thought of as a grouping of homogeneous rows, vertices, or edges. A VertexFrame may store all vertices representing a single entity type, while an EdgeFrame may store all edges representing a single type of relationship or connection. For example, in a social network graph, there might be one vertex frame representing people and several edge frames, each storing connections of a different type: “friend-of”, “knows”, “family”, “works-with”. A TableFrame is most often used to store results of a query.

Every frame carries metadata about its lifetime: the created_at and updated_at properties give the frame’s creation and last-update times as timezone-aware datetimes, or None when the time is unavailable.

In addition to these three data frames, xGT supports collecting vertex and edge frames into graphs. Graphs in xGT can be used in whole-graph algorithms and queries. The GraphFrame class implements this concept. See Graphs for more details.

Creating and dropping frames is discussed in detail in Frame Management.

2.2.2. Namespaces

All frames in xGT are grouped into namespaces and frame names need to be unique only within a namespace. The name of a namespace must be globally unique within xGT. Namespaces can be thought of as living under a global namespace directory that xGT keeps for its users.

By default, each user has a private namespace with the same name as their userid. Only the user has access to see frames in their private namespace. The sysadmin can turn off private namespaces by setting the config value system.auto_create_user_namespace to false. If logging in without a userid, the private namespace for the session is MajorTom.

A user might be able to do all their work within their private or other existing namespaces. If a new namespace is needed, it can be created with create_namespace():

server.create_namespace('mynamespace')

If the namespace indicated by a frame name does not already exist, it will be created along with the frame. In this example, if the namespace named graph does not already exist, it will be created so that it can store graph__VertexFrame.

vertex_frame = server.create_vertex_frame(name = 'graph__VertexFrame',
                                          schema = [['id', xgt.INT]],
                                          key = 'id')

Client API calls can be used to obtain the list of namespaces with get_namespaces() and delete a namespace with drop_namespace(). Unless the force_drop parameter is set to true, a namespace will not be dropped if it contains any frames.

2.2.2.1. Default Namespace

To simplify usage, xGT supports the use of a default namespace. When a connection to the server is created, the default namespace is set to the user’s private namespace. If the sysadmin has set the config value system.auto_create_user_namespace to false, then the default namespace is initially disabled and a user must explicitly set it. (See Disabling the Default Namespace.)

Any API calls that do not specify a namespace will occur in the default namespace. For example, consider user userA with the private namespace userA. If the default is not changed, any API call that refers to FrameA will be translated to userA__FrameA. In a single user environment or for any situation where namespaces are not needed, users can ignore the existence of namespaces and perform all actions within their default namespace. Users can then simply refer to frames by their non fully qualified names. For simplicity, most examples in the documentation will assume that frames are in the default namespace.

The default namespace can be changed with set_default_namespace(). This only changes the default namespace for the current user session. The current default namespace can be obtained with get_default_namespace().

A user can change the default namespace for a connection object any number of times. A Python script may use multiple connection objects, each with their own default namespace. The example below uses two connection objects to create a frame career__Employees and to retrieve frames career__Companies, cyber__Devices, and cyber__Netflow.

conn1.set_default_namespace('career')
conn2.set_default_namespace('cyber')
conn1.create_vertex_frame(name = 'Employees',
                          schema = [['person_id', xgt.INT],
                                    ['name', xgt.TEXT]],
                          key = 'person_id')
companies = conn1.get_frame(name = 'career__Companies')
netflow = conn1.get_frame(name = 'cyber__Netflow')
devices = conn2.get_frame(name = 'Devices')

Note in the example above that the namespace can either be provided or omitted. The default namespace of conn1 is career, so giving the fully qualified name career__Companies is redundant but allowed. Even though its default namespace is career, conn1 is used to retrieve the frame cyber__Netflow in the cyber namespace by providing the fully qualified name.

2.2.2.2. Disabling the Default Namespace

The default namespace can be disabled, which requires all API calls to provide the fully qualified name of any frame used. This is done by passing in either the empty string or None to set_default_namespace():

server.set_default_namespace(None)

Note that this only disables a default namespace for the given user session.

2.2.3. Graphs

xGT implements the concept of a graph as a collection of vertex and edge frames. The vertex and edge frames forming the graph can be in different namespaces. Graphs can be used to logically organize frames into a coherent single entity which can be used in queries and whole-graph algorithms.

Graphs can be used to call whole-graph algorithms operating on them without having to enumerate the individual frames (see Whole Graph Algorithms for details).

Graphs can be created by using the create_graph() method which returns an instance of the GraphFrame class:

people_graph = server.create_graph('PeopleGraph',
                                   { 'Person', 'FriendOf', 'WorksFor' })

For this example, the vertex frame Person and the edge frames FriendOf and WorksFor already exist on the server. The example creates a graph named “PeopleGraph” with the vertex frame “Person” and the edge frames “FriendOf” and “WorksFor” as members. The members of the graph can be specified as a set, as in the previous example, or as a dictionary mapping aliases to the constituent frames:

people_graph = server.create_graph('PeopleGraph', { 'p' : 'Person',
                                                    'f' : 'FriendOf',
                                                    'w' : 'WorksFor' })

The get_graphs() method returns a list of the names of the graphs present on the server that the user has access to subject to Access Control.

The get_frame() method can be used to return an instance of an existing graph:

people_graph = server.get_frame('PeopleGraph')

Graph frames can be removed from the system by calling the method drop_graph on the connection object and passing in the graph frame instance or its name:

server.drop_graph('PeopleGraph')
other_graph = server.get_frame('OtherGraph')
server.drop_graph(other_graph)

Dropping a graph removes only the graph itself; the member frames and the data they hold are left untouched.

Vertex and edge frames have a graphs property that lists the graphs that the frame belongs to, subject to access control.

person = server.get_frame('Person')
print(person.graphs)

2.2.3.1. Graph Frame

The GraphFrame class provides an interface to existing graphs on the xGT server. GraphFrames are similar to namespaces in the sense that they store references to other frames inside them. In contrast, vertex, edge and table frames store user data inside them. Like the other frames, graph frames exist in a namespace. Multiple graph frames can exist in a namespace, as long as their names are unique within that namespace.

The GraphFrame class provides properties and methods to retrieve the current members of a graph, as well as to add and remove vertex and edge frames.

people_graph = server.get_frame('PeopleGraph')
print(people_graph.members)
people_graph.remove_members({ 'WorksFor' })
people_graph.add_members({ 'w' : 'WorksFor' })

Adding and removing members changes only the graph’s membership; the member frames and the data they hold are never modified. To remove every member from the graph in a single atomic operation, use the remove_all_members() method.

Members can also be added by passing in instances of vertex or edge frames:

people_graph = server.get_frame('PeopleGraph')
works_for = server.get_frame('otherns__WorksFor')
people_graph.add_members({ works_for })

In this example, we add an edge frame from a different namespace to the graph. The edge frame is also added using a set meaning it has no alias. When a frame is added without an alias, it is automatically given an alias of its fully qualified name. For this example, the alias for works_for is “otherns__WorksFor”.

Adding an edge frame to a graph will also add its source and target vertex frames. Removing a vertex frame from a graph also removes any dependent edge frames in the graph.

The members property returns frame names. To work with the members as frame objects, use the get_member_frames() method, which returns a dictionary mapping aliases to VertexFrame and EdgeFrame objects. The membership and the member frames are resolved together, so the result is a consistent snapshot of the graph. Passing ‘vertex’ or ‘edge’ as the frame_type parameter restricts the result to members of that type:

for alias, frame in people_graph.get_member_frames('vertex').items():
  print(alias, frame.num_vertices)

The member frames of a graph can also be retrieved directly from the connection with the graph parameter of get_frames(), which composes with its frame_type parameter:

member_vertex_frames = server.get_frames(graph = 'PeopleGraph',
                                         frame_type = 'vertex')

Like the other frame types, graphs carry metadata: the created_at and updated_at properties give the creation and last-update times, and the num_vertices and num_edges properties give the total counts across the member frames.

2.2.3.2. Clearing a Graph

The clear() method removes all rows from every member frame, keeping the frames and their schemas. All member frames are cleared in a single transaction, so a member vertex frame and its member incident edge frames are emptied together. The graph itself and its membership are left intact. If a member vertex frame has a non-empty incident edge frame that is not a member of the graph, the clear fails and no frame is cleared.

Note that clearing a graph affects the data held by the member frames, while remove_all_members() affects only the graph’s membership:

people_graph.clear()               # Removes the rows of every member frame.
people_graph.remove_all_members()  # Removes every frame from the graph.

2.2.3.3. Graph Frames and Queries

Aliases of graph frames can be used as type annotations instead of frame names in queries when the graph has been set as the default graph:

people_graph = server.create_graph('PeopleGraph', { 'p' : 'Person',
                                                    'f' : 'FriendOf',
                                                    'w' : 'WorksFor' })
server.set_default_graph(people_graph)

The following two queries are equivalent:

MATCH (v:p)-[:f]->(:w)
RETURN count(*)

MATCH (v:Person)-[:FriendOf]->(:WorksFor)
RETURN count(*)

To disable the default graph, set it to None: set_default_graph(None). There is a complementary call on the connection object to obtain the current default graph: get_default_graph(). It will return the current default graph as a GraphFrame instance or None if there is no default graph.

2.2.4. Access Control

xGT supports access control on frames. There are two categories of access control: frame and row. Frame security protects access to the entire frame: a user is either granted a certain type of access to all rows in a frame or not. Row security protects individual rows of a frame: vertices in a vertex frame, edges in an edge frame, and rows in a table frame. A frame may have no access control, only frame access control, only row access control, or both. Currently, row access control is not supported for namespaces.

More information about using xGT with access control is found in Access Control, and information about setting up and managing access control is found in Access Control.

2.2.4.1. Frame Access Control

When creating frames or namespaces, security labels can optionally be attached for each of the access types:

• Create: The create access type allows the insertion of new rows into the frame.

• Read: The read access type allows reading operations on the data rows held by the frame.

• Update: The update access type allows the updating of existing rows in the frame. Columns of existing rows can be modified if this access type is granted.

• Delete: The delete access type allows the deletion of existing rows from a frame. Delete access on a frame is also required to delete the frame itself.

In order to perform an action on a frame or namespace, the connected user must have required labels in their label set. For example, to read a frame, the user must have the labels attached to read access for that frame. To load new data into the frame, the user must have the labels attached to read access and create access for that frame. If xGT is being used with access control, the label set of a user will be configured by the administrator as described in Configuring Groups and Labels.

Security labels are attached to a frame by passing into a frame creation method the dictionary parameter frame_labels that contains a key for each of the access types:

namespace_labels = { 'create' : ['label1'],
                     'delete' : ['label1']
                   }
server.create_namespace(name = 'graph',
                        frame_labels = namespace_labels)
frame_labels = { 'create' : ['label1', 'label2'],
                 'read'   : ['label2'],
                 'update' : ['label1', 'label2', 'label3'],
                 'delete' : ['label1', 'label2', 'label3']
               }
vertex_frame = server.create_vertex_frame(name = 'graph__VertexFrame',
                                          schema = [['id', xgt.INT]],
                                          key = 'id',
                                          frame_labels = frame_labels)

The above example creates a namespace with one set of labels and a frame within it with a different set of labels. If a namespace does not exist and is implicitly created along with a frame, it will be assigned the same labels as the frame. By default, no labels are attached for any of the four access types. Any attempts to perform an action for which the user does not have the required label set will result in an XgtSecurityError() being thrown.

Graphs accept the same frame_labels parameter through create_graph(). A user must have the graph’s read labels to see or use the graph. The labels on a graph are independent of the labels on its member frames; access to a graph does not grant access to the data in its members. The user_permissions property reports which actions on a graph the connected user is allowed to perform.

2.2.4.2. Row Access Control

While frame access control restricts access to the entire frame, row access control protects individual rows in a frame. The word “row” is used to mean individual vertices in a vertex frame, edges in an edge frame, and rows in a table frame. Security on a row is binary: an authenticated user can either access the row or not. If a user does not have access to a row, they will interact with the frame as though the row did not exist. For instance, it will not be accessed during an OpenCypher query or be viewable during an egest.

Row access control is provided by attaching security labels to individual rows. This occurs when a row is created: either during ingest with load() or insert() or during an OpenCypher query that creates new vertices or edges. Section Setting Row Labels provides more information on attaching security labels to rows during ingest. An authenticated user will only be able to view or access rows whose security labels are a subset of the user’s labels. Currently, row access control is not supported for namespaces, only for vertex, edge and table frames.

Security labels can only be attached to rows if the frame is created with row access control. To create such a frame, the create frame method should be called with the parameter row_label_universe equal to a list of all possible security labels that can be attached to any future row in that frame:

row_labels = [ 'label1', 'label2', 'label3', 'label4', 'label9' ]
vertex_frame = server.create_vertex_frame(name = 'graph__VertexFrame',
                                          schema = [['id', xgt.INT]],
                                          key = 'id',
                                          row_label_universe = row_labels)

These labels must exist in the xgt__Label frame, as described in Configuring Groups and Labels. Note that the Row Label Universe, which is the set of possible security labels for a row in this frame, cannot be changed after frame creation. The maximum size of this row label universe for any one frame is 128 labels.

The row label universe attached to a frame can be retrieved with row_label_universe. However, this will return only those labels that are both in that frame’s row label universe and in the authenticated user’s label set. If the frame has additional labels in its row label universe that are not visible to the user, they will not be returned.