QNX series: v. resource manager

The Bones of a Resource Manager

This article will describe the general framework and layering from both server and client sides, and give examples.

1. Under the covers

1.1 Under the client's covers

When a client calls a function that requires path name resolution (such as open()/rename()/stat()/unlink()), it will send a message to the process manager and the corresponding resource manager at the same time, so as to obtain the file descriptor and send a message to the device associated with the path name through the file descriptor.

/*
 * In this stage, the client talks 
 * to the process manager and the resource manager.
 */
fd = open("/dev/ser1", O_RDWR);

/*
 * In this stage, the client talks directly to the
 * resource manager.
 */
for (packet = 0; packet < npackets; packet++)
{
    write(fd, packets[packet], PACKET_SIZE);
}
close(fd);

How is the logic behind the above code implemented? Let's assume that this is a serial port device managed by the devc-ser8250 resource manager, and the registered path name is dev/ser1:

Under-the-cover communication between the client, the process manager, and the resource manager

  1. When the client sends a query message, the open() function will send a message to the process manager requesting to find the name, such as / dev/ser1;
  2. The process manager returns the nd/pid/child/handle associated with the pathname. When the devc-ser8250 resource manager registers with / dev/ser1, the process manager will maintain an entry information similar to the following:
    0, 47167, 1, 0, 0, /dev/ser1
    Each item of the entry represents:
  • 0, Node descriptor (nd), used to describe nodes in the network;
  • 47167, Process ID (pid), Process ID number of resource manager;
  • 1. Channel ID (chid), the channel used by the resource manager to receive messages;
  • 0, the index number of the Handle, the Handle registered by the resource manager;
  • 0, the opening type passed during name registration;
  • /dev/ser1, registered pathname;
    An entry table is maintained in the process manager to record the information of each resource manager. The longest match will be selected when the name matches.
  1. The client needs to send a connect message to the resource manager. First, it needs to create a connection channel:
fd = ConnectAttach(nd, pid, chid, 0, 0);

The client also uses this fd to send a connection message to the resource manager, requesting to open / dev/ser1. After receiving the message, the resource manager will check the permissions.

  1. The resource manager usually responds with a pass or fail.
  2. When the client obtains the file descriptor, it can send a message to the device directly through it. The sample code looks like the client writes directly to the device. In fact, when calling write(), it will send a message first_ IO_ The write message is sent to the resource manager to request data writing. When the client calls close(), it will be sent_ IO_ CLOSE_ The DUP message is sent to the resource manager to complete the final resource recycling and cleaning.

1.2 Under the resource manager's covers

The resource manager is a server that receives and replies to messages through the send/receive/reply message protocol. The pseudo code is as follows:

initialize the resource manager
register the name with the process manager
DO forever
    receive a message
    SWITCH on the type of message
        CASE _IO_CONNECT:
            call io_open handler
            ENDCASE
        CASE _IO_READ:
            call io_read handler
            ENDCASE
        CASE _IO_WRITE:
            call io_write handler
            ENDCASE
        .   /* etc. handle all other messages */
        .   /* that may occur, performing     */
        .   /* processing as appropriate      */
    ENDSWITCH
ENDDO

In fact, many details in the above code may not be used in the implementation of a resource manager, because some libraries are encapsulated, but you still need to implement the reply to the message.

2. Layers in a resource manager

The resource manager consists of four layers, from bottom to top:

  1. iofunc layer
  2. resmgr layer
  3. dispatch layer
  4. thread pool layer

2.1 iofunc layer

The main function of this layer is to provide POSIX features. When writing a resource manager, users don't need to pay much attention to the details involved in providing POSIX file system to the outside world.
This layer consists of a set of functions iofunc_* Composition, including the default handler. If you do not provide your own handler, the default provider will be used. For example, if IO is not provided_ If the open handler, iofunc will be called_ open_ default(). It also contains the help functions called by the default handler. When you implement the handler, you can also call these help functions.

2.2 resmgr layer

This layer is responsible for managing most of the details of the Explorer Library:

  • Check incoming messages

  • Call the appropriate handler to process the message
    If you do not use this layer, you need to parse the message yourself, which is used by most resource managers.

    The resmgr layer to handle IO* message

2.3 dispatch layer

This layer acts as a blocking point for multiple event types. Using this layer, you can handle:

  • `IO * message, using resmgr layer;

  • select(), which registers a processing function and is called when the packet arrives. The function here is select_* () function;

  • Pulses: register a processing function and call it when pulses come. The function here is pulse_* () function;

  • For other messages, you can provide a series of message types and processing functions created by yourself. When the message arrives, call the corresponding processing function. The function here is

    message_*()
    

    Function;

    use the dispatch layer to handle IO* messages, select, pulses, and other messages

    If there is no matching message, the Handler function will be called to find the matching message, and then

    MsgError
    

2.4 thread pool layer

This means that a thread manager can create a single thread or multiple threads. The thread needs to be provided with blocking function and the processing function to be called when the blocking function returns. In most cases, the task of the dispatch layer can be assigned to the thread. Of course, it can also be a resmgr layer task or a function implemented by itself.

3. Simple examples of device resource managers

3.1 Single-threaded device resource managers

Let's first look at a complete single threaded device resource manager code:

#include <errno.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/iofunc.h>
#include <sys/dispatch.h>

static resmgr_connect_funcs_t    connect_funcs;
static resmgr_io_funcs_t         io_funcs;
static iofunc_attr_t             attr;

main(int argc, char **argv)
{
    /* declare variables we'll be using */
    resmgr_attr_t        resmgr_attr;
    dispatch_t           *dpp;
    dispatch_context_t   *ctp;
    int                  id;

    /* initialize dispatch interface */
    if((dpp = dispatch_create()) == NULL) {
        fprintf(stderr,
                "%s: Unable to allocate dispatch handle.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize resource manager attributes */
    memset(&resmgr_attr, 0, sizeof resmgr_attr);
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* initialize functions for handling messages */
    iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                     _RESMGR_IO_NFUNCS, &io_funcs);

    /* initialize attribute structure used by the device */
    iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

    /* attach our device name */
    id = resmgr_attach(
            dpp,            /* dispatch handle        */
            &resmgr_attr,   /* resource manager attrs */
            "/dev/sample",  /* device name            */
            _FTYPE_ANY,     /* open type              */
            0,              /* flags                  */
            &connect_funcs, /* connect routines       */
            &io_funcs,      /* I/O routines           */
            &attr);         /* handle                 */
    if(id == -1) {
        fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
        return EXIT_FAILURE;
    }

    /* allocate a context structure */
    ctp = dispatch_context_alloc(dpp);

    /* start the resource manager message loop */
    while(1) {
        if((ctp = dispatch_block(ctp)) == NULL) {
            fprintf(stderr, "block error\n");
            return EXIT_FAILURE;
        }
        dispatch_handler(ctp);
    }
}

The above code completes the following functions:

  1. Initialize the dispatch interface
    By calling dispatch_create() interface to create a dispatch_t structure, which contains the channel ID, but the channel ID is actually created when attaching other contents, such as calling resmgr_attach()/message_attach()/pulse_attach() etc.
  2. Initialize Explorer properties
    When resmgr is called_ Resmgr is passed in when attach()_ attr_ T structure. In this example, the main configurations are:
  • nparts_max, the number of IOV s that can be replied by the server;
  • msg_max_size, maximum receive buffer size;
  1. Initialize message handling function
    In this example, two tables are provided to specify which function to call when a specific message arrives:
  • Connection function table;
  • I/O function table;
    You can call iofunc_func_init() interface to configure the default operation function.
  1. Initializes the attribute structure used by the device
    Call iofunc_attr_init() interface. The attribute structure should at least include the following information:
  • Permissions and device types
  • Group ID and owner ID
  1. Register a name in the namespace
    Call resmgr_attach() to register the path of the resource manager. Before the resource manager can receive messages from other programs, it needs to notify other programs of its binding relationship with pathnames through the process manager.
  2. Assign context structure
    Call dispatch_context_alloc() interface. The context structure contains buffers for receiving messages and IOVs buffers for replying to messages.
  3. Start resource manager message loop
    After entering the loop, the resource manager will be in the dispatch_ Receive the message in block () and call dispatch_handler() to distribute and select the appropriate functions in the connection function table and I/O function table for execution. When the execution is completed, it will enter the dispatch again_ Block () to wait for other messages to be received.

3.2 Multi-threaded device resource managers

The following is the sample code of multithreaded device resource manager:

#include <errno.h>
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
#include <unistd.h>

/*
 * Define THREAD_POOL_PARAM_T such that we can avoid a compiler
 * warning when we use the dispatch_*() functions below
 */
#define THREAD_POOL_PARAM_T dispatch_context_t

#include <sys/iofunc.h>
#include <sys/dispatch.h>

static resmgr_connect_funcs_t    connect_funcs;
static resmgr_io_funcs_t         io_funcs;
static iofunc_attr_t             attr;

main(int argc, char **argv)
{
    /* declare variables we'll be using */
    thread_pool_attr_t   pool_attr;
    resmgr_attr_t        resmgr_attr;
    dispatch_t           *dpp;
    thread_pool_t        *tpp;
    dispatch_context_t   *ctp;
    int                  id;

    /* initialize dispatch interface */
    if((dpp = dispatch_create()) == NULL) {
        fprintf(stderr, "%s: Unable to allocate dispatch handle.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize resource manager attributes */
    memset(&resmgr_attr, 0, sizeof resmgr_attr);
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* initialize functions for handling messages */
    iofunc_func_init(_RESMGR_CONNECT_NFUNCS, &connect_funcs, 
                     _RESMGR_IO_NFUNCS, &io_funcs);

    /* initialize attribute structure used by the device */
    iofunc_attr_init(&attr, S_IFNAM | 0666, 0, 0);

    /* attach our device name */
    id = resmgr_attach(dpp,            /* dispatch handle        */
                       &resmgr_attr,   /* resource manager attrs */
                       "/dev/sample",  /* device name            */
                       _FTYPE_ANY,     /* open type              */
                       0,              /* flags                  */
                       &connect_funcs, /* connect routines       */
                       &io_funcs,      /* I/O routines           */
                       &attr);         /* handle                 */
    if(id == -1) {
        fprintf(stderr, "%s: Unable to attach name.\n", argv[0]);
        return EXIT_FAILURE;
    }

    /* initialize thread pool attributes */
    memset(&pool_attr, 0, sizeof pool_attr);
    pool_attr.handle = dpp;
    pool_attr.context_alloc = dispatch_context_alloc;
    pool_attr.block_func = dispatch_block; 
    pool_attr.unblock_func = dispatch_unblock;
    pool_attr.handler_func = dispatch_handler;
    pool_attr.context_free = dispatch_context_free;
    pool_attr.lo_water = 2;
    pool_attr.hi_water = 4;
    pool_attr.increment = 1;
    pool_attr.maximum = 50;

    /* allocate a thread pool handle */
    if((tpp = thread_pool_create(&pool_attr, 
                                 POOL_FLAG_EXIT_SELF)) == NULL) {
        fprintf(stderr, "%s: Unable to initialize thread pool.\n",
                argv[0]);
        return EXIT_FAILURE;
    }

    /* start the threads; will not return */
    thread_pool_start(tpp);
}

As you can see from the code, most of the code is the same as the single threaded example. In this code, the thread uses dispatch in the blocking loop_* () function (dispatch layer).

  1. Initialize thread pool properties
    Give pool_ The assignment of each field of attr is used to tell the thread which functions to call when blocking the loop and how many threads the thread pool needs to maintain.
  2. Allocate the handle of a thread pool
    Call thread_pool_create() interface, which is used to control the whole thread pool.
  3. Open thread
    Call thread_pool_start() interface to start the thread pool. Each newly created thread will use the context given in the attribute structure_ Alloc() function to allocate THREAD_POOL_PARAM_T defines the context structure of the type.

3.3 Using MsgSend() and MsgReply()

Instead of using the read() and write() interfaces to interact with the resource manager, you can call MsgSend() to send messages. The following example will be divided into two parts: server and client. The server side needs to enable PROCMGR_AID_PATHSPACE, which is used to ensure that resmgr can be called_ Attach() function.

  • Server code:
/*
 * ResMgr and Message Server Process
 */

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <sys/neutrino.h>
#include <sys/iofunc.h>
#include <sys/dispatch.h>

resmgr_connect_funcs_t  ConnectFuncs;
resmgr_io_funcs_t       IoFuncs;
iofunc_attr_t           IoFuncAttr;


typedef struct
{
    uint16_t msg_no;
    char     msg_data[255];
} server_msg_t;


int message_callback( message_context_t *ctp, int type, unsigned flags, 
                      void *handle )
{
    server_msg_t *msg;
    int num;
    char msg_reply[255];

    /* Cast a pointer to the message data */
    msg = (server_msg_t *)ctp->msg;

    /* Print some useful information about the message */
    printf( "\n\nServer Got Message:\n" );
    printf( "  type: %d\n" , type );
    printf( "  data: %s\n\n", msg->msg_data );

    /* Build the reply message */
    num = type - _IO_MAX;
    snprintf( msg_reply, 254, "Server Got Message Code: _IO_MAX + %d", num );
   
    /* Send a reply to the waiting (blocked) client */ 
    MsgReply( ctp->rcvid, EOK, msg_reply, strlen( msg_reply ) + 1 );

    return 0;
}



int main( int argc, char **argv )
{
    resmgr_attr_t        resmgr_attr;
    message_attr_t       message_attr;
    dispatch_t           *dpp;
    dispatch_context_t   *ctp, *ctp_ret;
    int                  resmgr_id, message_id;

    /* Create the dispatch interface */
    dpp = dispatch_create();
    if( dpp == NULL )
    {
        fprintf( stderr, "dispatch_create() failed: %s\n", 
                 strerror( errno ) );
        return EXIT_FAILURE;
    }

    memset( &resmgr_attr, 0, sizeof( resmgr_attr ) );
    resmgr_attr.nparts_max = 1;
    resmgr_attr.msg_max_size = 2048;

    /* Setup the default I/O functions to handle open/read/write/... */
    iofunc_func_init( _RESMGR_CONNECT_NFUNCS, &ConnectFuncs,
                      _RESMGR_IO_NFUNCS, &IoFuncs );

    /* Setup the attribute for the entry in the filesystem */
    iofunc_attr_init( &IoFuncAttr, S_IFNAM | 0666, 0, 0 );

    resmgr_id = resmgr_attach( dpp, &resmgr_attr, "serv", _FTYPE_ANY, 
                               0, &ConnectFuncs, &IoFuncs, &IoFuncAttr );
    if( resmgr_id == -1 )
    {
        fprintf( stderr, "resmgr_attach() failed: %s\n", strerror( errno ) );
        return EXIT_FAILURE;
    }

    /* Setup our message callback */
    memset( &message_attr, 0, sizeof( message_attr ) );
    message_attr.nparts_max = 1;
    message_attr.msg_max_size = 4096;

    /* Attach a callback (handler) for two message types */
    message_id = message_attach( dpp, &message_attr, _IO_MAX + 1,
                                 _IO_MAX + 2, message_callback, NULL );
    if( message_id == -1 )
    {
        fprintf( stderr, "message_attach() failed: %s\n", strerror( errno ) );
        return EXIT_FAILURE;
    }

    /* Setup a context for the dispatch layer to use */
    ctp = dispatch_context_alloc( dpp );
    if( ctp == NULL )
    {
        fprintf( stderr, "dispatch_context_alloc() failed: %s\n", 
                 strerror( errno ) );
        return EXIT_FAILURE;
    }


    /* The "Data Pump" - get and process messages */
    while( 1 )
    {
        ctp_ret = dispatch_block( ctp );
        if( ctp_ret )
        {
            dispatch_handler( ctp );
        }
        else
        {
            fprintf( stderr, "dispatch_block() failed: %s\n", 
                     strerror( errno ) );
            return EXIT_FAILURE;
        }
    }

    return EXIT_SUCCESS;
}
  1. The first is to call dispatch_create() interface to create dpp, and forward the received message to the appropriate layer (resmgr, message, pulse) through this handle;
  2. Set call resmgr_ Variables required by attach();
  3. Call iofunc_func_init() to set the default processing function and call iofunc_attr_init() to set the attribute structure;
  4. Call resmgr_attach(). Note that the registered path is not an absolute path at this time, so it is under its execution path by default
  5. Tell the dispatch layer that it needs to process its own messages in addition to the standard I/O and connection messages processed by the resmgr layer.
  6. Call message_ The attach () interface to register its own message handling function.
  7. When dispatch is called_ Block() calls dispatch after receiving the message_ Handler (), which will actually be processed in dispatch_ Calling message in handler()_ Callback() function. When the message type is_ IO_MAX + 1 or_ IO_ When MAX + 2, the function will be called back.
  • Client code:
/* 
 * Message Client Process 
 */ 

#include <stdio.h> 
#include <unistd.h> 
#include <stdlib.h> 
#include <fcntl.h> 
#include <errno.h> 
#include <string.h> 
#include <sys/neutrino.h> 
#include <sys/iofunc.h> 
#include <sys/dispatch.h> 

typedef struct 
{ 
    uint16_t msg_no; 
    char msg_data[255]; 
} client_msg_t; 

int main( int argc, char **argv ) 
{ 
    int fd; 
    int c; 
    client_msg_t msg; 
    int ret; 
    int num; 
    char msg_reply[255]; 
    
    num = 1; 
    
    /* Process any command line arguments */ 
    while( ( c = getopt( argc, argv, "n:" ) ) != -1 ) 
    { 
        if( c == 'n' ) 
        { 
            num = strtol( optarg, 0, 0 ); 
        } 
    } 
    /* Open a connection to the server (fd == coid) */ 
    fd = open( "serv", O_RDWR ); 
    if( fd == -1 ) 
    { 
        fprintf( stderr, "Unable to open server connection: %s\n", 
            strerror( errno ) ); 
        return EXIT_FAILURE; 
    } 
    
    /* Clear the memory for the msg and the reply */ 
    memset( &msg, 0, sizeof( msg ) ); 
    memset( &msg_reply, 0, sizeof( msg_reply ) ); 
    
    /* Set up the message data to send to the server */ 
    msg.msg_no = _IO_MAX + num; 
    snprintf( msg.msg_data, 254, "client %d requesting reply.", getpid() ); 
    
    printf( "client: msg_no: _IO_MAX + %d\n", num ); 
    fflush( stdout ); 
    
    /* Send the data to the server and get a reply */ 
    ret = MsgSend( fd, &msg, sizeof( msg ), msg_reply, 255 ); 
    if( ret == -1 ) 
    { 
        fprintf( stderr, "Unable to MsgSend() to server: %s\n", strerror( errno ) ); 
        return EXIT_FAILURE; 
    } 
    
    /* Print out the reply data */ 
    printf( "client: server replied: %s\n", msg_reply ); 
    
    close( fd ); 
    
    return EXIT_SUCCESS; 
} 

The client obtains the connection id through the open() interface, calls the MsgSend() interface to send a message to the server, and waits for a reply.

The above examples of server and client are very simple, but cover most of the content. Based on this basic framework, other things can be done:

  • Register different message callback functions based on different message types;
  • In addition to messages, use pulse_attach() to receive pulse;
  • Rewrite the default I/O message processing function so that the client can also use read() and write() to interact with the server;
  • Using thread pool to write multithreaded server;

Posted by meepokman on Mon, 02 May 2022 02:57:17 +0300