| iMatix home page
| Xitami home page
| << | < | > | >> 		Xitami Xitami
Version 2.4d7
 

Extending Xitami with External Peer Processes

By Robin Dunn

Introduction

Any serious web-based application relies, possibly quite heavily, on dynamic content. In other words, dynamically generated pages.

CGI

To facilitate the need for dynamic content, CGI was invented back in the dark-ages of the Web (just a few years ago.) While CGI provides the ability to do dynamic generation of web pages as well as quite an elegant and flexible interface, it also has a very big wart. It is too slow for serious work. The basic model that CGI follows is:

  1. The web server receives a request that maps to a CGI handler.
  2. The web server starts a new process with the specified CGI command, feeding the CGI data to it.
  3. The web server waits for the process to terminate.
  4. The output of the process is sent to the browser.

Unfortunately, each time the CGI process is started, it can be several seconds before it is able to even begin processing the request. Depending on the OS there may be a large amount of overhead in creating a new process. There may be a Perl or other language interpreter to load and initialize, followed by loading and parsing the CGI script itself. Database connections may have to be made, files opened, etc.

Server Extensions

Because of the shortcomings of CGI, server extension APIs were created by the various web server vendors. For example, Netscape provides NSAPI, Microsoft ISAPI, Apache ASAPI, and even Xitami has WSX. (Isn't it refreshing to not have "SAPI" at the end?)

While these extension APIs do away with the overhead of process creation they introduce a few warts of their own:

  1. The extension is linked into the server itself, (either statically at compile/link time or dynamically at runtime) so bugs in the extension can potentially bring down the entire web-server.
  2. Server extensions by their nature are highly specific to the API it is written for. It would be very difficult to be portable to another server.
  3. You are limited in your choice of languages to what is supported by the API.

Persistent CGI

There have been several alternatives that combine the ease of use and flexibility of CGI with the efficiency of Server extension APIs. One of the most popular of these is called FastCGI developed by Open Market, Inc. FastCGI or other persistent CGI alternatives are, as the name suggests, a CGI type of interface that is Faster than CGI. It could potentially be up to 20 times faster, or more, depending on what kind of startup/initialization overhead the script has. It does this phenomenal feat by allowing the CGI process to handle more than a single request each time it starts up. The web server communicates with the running persistent CGI process via a socket and sends the appropriate data to the persistent CGI process when the reqest is made. The persistent CGI process does its thing, generates an html page or other response and sends it back over the socket and then instead of terminating, it waits for the next request.

LRWP

Begining with version 2.2a, Xitami includes a persistent CGI extension called LRWP, (which stands for Long Running Web Process.) LRWP is written as a WSX agent which implements a simple protocol for communicating with external processes called Peers. The peer process simply waits for the requests to come from Xitami and then responds with a valid http response, just like a CGI program. The main difference is that like other persistent CGI solutions, the LRWP peer waits around for another HTTP request instead of terminating.

LRWP is not FastCGI, but it is very similar. The protocol between Xitami and the LRWP Peer is much simpler than the FastCGI protocol, and therefore less prone to implementation differences or errors. However, given the right circumstances it is possible to run the same program as both a LRWP Peer and a FastCGI Peer with only minor differences in the startup code.

The LRWP protocol

As mentioned above, the communications protocol between the LRWP Peer and the LRWP Agent within Xitami is quite simple.

  1. The Peer connects to the port specified in the [LRWP] section of the Xitami configuration file, (offset by portbase if needed,) and sends a startup string. The startup string has three components:
    1. The appname or alias. This is what will be used to redirect HTTP requests to the Peer. For example, if you use "testapp" as the alias, then requests begining with http://yourhostname.com/testapp will be sent to the peer. If the alias is "this/is/a/test" then requets begining with http://yourhostname.com/this/is/a/test will be sent.

      If multiple peers connect with the same alias name, then multiple simultaneous requests will be able to be handled. If there are more simultaneous requests then there are peers to handle them, then the LRWP Agent in Xitami will queue up the requests and wait for a peer to complete its current request.

    2. The next component of the startup string is the name of the virtual host to be associated with. If a name is passed, then the peer will only be given requests that are directed to that virtual host. Otherwise, the peer will be sent requests from any virtual or nonvirtual host that matches the alias.
    3. The third component of the startup string is reserved for a currently unimplemented feature.

    Each component of the startup string is separated by a character with an ASCII value of 255.

  2. The LRWP Agent then responds with either "OK" or an error message that begins with "ERROR". If "OK" then the peer is good to go, otherwise the LRWP Agent is rejecting the connection for some reason and you have some debugging to do.
  3. At this point the Peer simply waits for a request to be sent to it. When the request comes, it is formatted as follows:

    ENV size ENV data as a series of name=value pairs POST size POST data, if any

    1. A nine-digit, zero-filled number representing the size of the CGI-environment data. For example, "000001234".
    2. Immediately following the size is that number of bytes of data formatted as NAME=VALUE pairs, each separated by a NULL byte.
    3. The next component is another nine-digit, zero-filled number, this time representing the size of the POST data, if any.
    4. If there is any POST data, the specifed number of bytes comes next.
  4. The Peer now has everything it needs to handle the request, so at this point it should do whatever it needs to create a HTML document, or some other type of data to return. The Peer then sends back on the socket the number of bytes it will be sending, formatted as a nine-digit, zero-filled number, followed by that number of bytes of data.

    The content returned to the server will be treated just like the output from CGI programs, so content-type and other headers are significant.

  5. At this point the Peer should either exit or go back to wait for another request. If it exits, and if there are no other peers attached to the server with the same alias name, then that URI will fall-back to default Xitami handling. For example, if the URI doesn't match a real file or directory then it returns an access error, otherwise the physical file or directory will be processed normally.

    There may be situations where this is a highly desirable feature, for example if you are running a contest and only want to collect 100 entries, the peer could exit after collecting the desired amount and then Xitami will fall-back to a static document at the same URI that explains that the contest is now closed.

    Another reason to exit is to ensure that any memory leaks in your program or the interpreter it is built upon are released. In order for this to be effective you probably will want to have the ability to automatically restart your peer processes. (UPM is currently being modified to allow this type of functionality.) For example, I know people with scripts running in the FastCGI environment that only allow the processes to handle 50 requests and then they exit and get restarted by an external process monitor.

LRWP Environment Variables

LRWP programs receive the same environment variables as a CGI program. If you're in doubt as to which ones these are, either read this manual again or run the testcgi program supplied with Xitami.

The LRWP Library

Included with Xitami is a small library of C functions to assist in writing LRWP Peers. The source files are lrwplib.h and lrwplib.c and contain the following functions: 

char* lrwp_connect(LRWP* lrwp,        /* pointer to UNCONNECTED LRWP object */
                   char* appname,     /* Name or alias of Peer app         */
                   char* host,        /* hostname/IP address to connect to */
                   char* port,        /* string containing port number     */
                   char* vhost)       /* optional virtual hostname         */
Connects to the LRWP agent running in Xitami on host and port. Sends the given appname to use as the URI alias that will trigger requests to be sent to this peer. If vhost is given, this peer will only be sent requests origininating from that virtual host. This function assumes that the LRWP structure is uninitialized and clears it before use.

Returns NULL on success and a pointer to an error message otherwise. 

int  lrwp_accept_request(LRWP* lrwp)    /* pointer to CONNECTED LRWP object  */
This funation waits for and recieves a request from the LRWP agent, and populates the LRWP structure with the request data.

Returns 0 on success and -1 otherwise. 

int  lrwp_send_string(LRWP* lrwp,       /* pointer to CONNECTED LRWP object  */
                      char* st)         /* an ouput string                   */
This function appends a string to the response buffer. lrwp_finish_request() must be called to send the response back to Xitami.

Returns 0 on success and -1 otherwise. 

int  lrwp_send_data(LRWP* lrwp,         /* pointer to CONNECTED LRWP object  */
                    void* data,         /* pointer to a data buffer          */
                    size_t len)         /* size of the data buffer           */
Appends a buffer of (possibly binary) data of the specified size to the response buffer. lrwp_finish_request() must be called to send the response back to Xitami.

Returns 0 on success and -1 otherwise. 

int  lrwp_finish_request(LRWP* lrwp)    /* pointer to CONNECTED LRWP object  */
Completes the request by sending the response buffer back to Xitami and prepares the lwrp structure to receive another request.

Returns 0 on success and -1 otherwise. 

int  lrwp_close(LRWP* lrwp)             /* pointer to CONNECTED LRWP object  */
Closes the connection to Xitami.

Returns 0 on success and -1 otherwise.

A Simple Example Using C

#include "sfl.h"
#include "lrwplib.h"

void main()
{
    LRWP    lrwp;
    int     count = 0;
    int     err;
    char*   errMsg;
    char    buf[256];

    sock_init();
    errMsg = lrwp_connect(&lrwp, "hello", "localhost", "81", "");
    if (errMsg) {
        fprintf(stderr, "%s\n", errMsg);
        exit(1);
    }
        /* only handle 5 reqests, then exit */
    while (count < 5 && lrwp_accept_request(&lrwp) != -1) {
        count += 1;
        lrwp_send_string(&lrwp, "Content-type: text/html\r\n\r\n");
        lrwp_send_string(&lrwp,
                "<HTML><HEAD><TITLE>hello</TITLE></HEAD>\n<BOD

        sprintf(buf, "<H2>Hello from LRWP</H2>\nCount = %d", count);
        lrwp_send_string(&lrwp, buf);

        lrwp_send_string(&lrwp, "\n</BODY></HTML>\n");
        lrwp_finish_request(&lrwp);
    }
    lrwp_close(&lrwp);
    sock_term();
}

A Simple Example Using Python

from lrwplib import LRWP

def main():
    lrwp = LRWP('hello', 'localhost', 81)
    lrwp.connect()
    count = 0
    while count < 5:        # exit after servicing 5 requests
        count = count + 1
        req = lrwp.acceptRequest()
        req.out.write('Content-type: text/html\r\n\r\n')
        req.out.write('<HTML><HEAD><TITLE>hello</TITLE><')
        req.out.write('</HEAD>\n')
        req.out.write('<H2>Hello from LRWP</H2>\nCount = ' + str(count))
        req.out.write('\n</BODY></HTML>\n')
        req.finish()

    lrwp.close()

if __name__ == '__main__':
    main()
| << | < | > | >>
| Welcome To Xitami | Table Of Contents | Installing Xitami | Administration | Configuration | Using The Common Gateway Interface (CGI) | Using SSI and Filters | Image Maps | Virtual Hosts | The FTP Service | A Beginner's Guide | Writing Web Server Extension (WSX) Agents | Extending Xitami with External Peer Processes | FAQ | Technical Implementation | Getting Support | Release History | License Agreement 		iMatix
Copyright © 1996-99 iMatix Corporation