GENERIC DISTRIBUTED TRANSACTION MANAGER

--Final Project for CS461

Protocols and Libraries

User Manual

Protocols and Libraries

Project Overview

Motivation

Our discussion about the atomic transaction and its property of "all-or-nothing" was interesting. Now I am thinking about two additional issues:

1. Many existing implementations of transactions need support from operating system or language runtime system. Can we implement it with a group of library procedures? That is, if a programmer of some distributed system wants to add the transaction feature to his program, he doesn't have to re-write his code in another language, he just needs to add some function calls in his existing code and link the transaction libraries to his program.

2. Can such a library be used by different programmers, e.g. one for a distributed database system and one for a distributed graphics editor?

My final project for this course will be based on the above ideas.

System Structure

The project consists of two parts:

1. Two transaction libraries, libtrsv.a(transaction server) and libtrcl.a(transaction clerk). The libtrsv.a will be linked to user's server program and libtrcl.a will be linked to user's client program. Libtrsv.a provides functions which operate on local datas, and libtrcl.a provides functions which maintain the states for a transaction which is invoked locally and operates remote datum. Some of the functions in libtrcl.a will call the remote version of the functions in libtrsv.a. Transactions are implemented with buffered write-ahead logs and two-phase commit protocol. In a distributed application, there could be multiple clients and servers, the servers could be running heterogeneous systems.

2. A stub generator which makes it possible for the transaction manager to support different kinds of users. There must be a way for the transaction manager to record what actions are taken during a transaction because those actions are not executed immediately by the user’s server, instead they will be executed by the transaction server when the transaction is committed. Also, some actions will be re-done when system reboots. A way to do this is to write in the log entry a function to call for executing or redoing the action and the argument to be passed to the function. The user is responsible for deciding which actions are to be delayed till the transaction commits and giving the primitive functions for those actions to the stub generator. The log writer can NOT just write a function pointer to the stable storage because the pointer will not be valid when the system reboots. It's the stub generator's job to solve this problem. It also saves the user from writing more code.

See Figure 1 for Part 1 and Figure 2 for Part 2.

Before continuing to read the rest of this document, please read the User Manual first for understanding the functions and files mentioned below.

Protocols and Libraries

Transaction Clerk – Two-phase Commit Protocol

The user client uses transactions as if the transactions are executed locally instead of on several remote servers. The transaction primitives available to user client are:

1. startClTransaction

2. endClTransaction

3. abortClTransaction

4. the transact_remote_transact_ version of any primitive actions defined by the user in tpc_func_list , e.g. transact_remote_transact_myWrite , transact_remote_transact_myInsert, transact_remote_transact_myDelete, etc.

For details about these functions, see "Function Definitions and Calls" in User Manual.

When a transaction is started on the client side, a new transaction record is added to the end of the local transaction clerk’s transaction queue. In the record, there is a list of the servers which are contacted during the transaction. It is empty initially.

When a user-defined primitive action is called for, the remote server who has the data which will be operated on in this action must be specified by both name and connection. They are passed to the function transact_remote_transact_<PFN>(PFN stands for Primitive Function Name) as arguments. The transaction clerk does NOT provide naming service because it is application-specific . The clerk first checks if the server is contacted within this transaction for the first time and adds the server to the servers’ list of this transaction if it is contacted for the first time. Then the clerk calls the remote_transact_ version of this primitive actions, which will be received and handled at the server side. The transact_remote_transact_ function returns 1 if the RPC is successful otherwise it returns 0.

The user will abort a transaction either because one of the primitive actions is unsuccessful or for any other reason. When a transaction is aborted, the clerk sends an ‘abort’ message to each server in the servers’ list by RPC.

When a transaction is terminated, the clerk first sends a ‘prepare’ message to each server in the servers’ list by RPC. If at least one of the RPCs does not return successfully, which means that at least one of the servers is not ready to commit the transaction, the clerk will abort the transaction automatically by sending an ‘abort’ message to each server. If all the RPCs for ‘prepare’ messages return successfully, the clerk writes a ‘finish’ entry to the log in stable storage together with the names of all the servers in the list. Then it sends a ‘commit’ message to each server by RPC. For those servers who didn’t return the RPCs successfully, the clerk keeps re-connecting to them and sending ‘commit’ messages by RPCs . After all RPCs return successfully, the clerk writes a ‘finish’ entry to the log.

Transaction Server – Write-ahead Log

A server consists of a thread for handling RPCs from clients, a user server and a transaction server. The possible RPCs that the server thread will handle are:

1. abortSvTransaction

2. prepareSvTransaction

3. commitSvTransaction

4. the transact_ version of any primitive actions defined by the user in tpc_func_list , e.g. transact_myWrite, transact_myInsert, transact_myDelete, etc.

5. any other functions specified by the user in user.x.

For details about these functions, see "Function Definitions and Calls" in User Manual.

When a user-defined primitive action is called for, the transaction server first checks if the transaction is a new one, i.e. if the server is contacted for this transaction for the first time. If it is, the server searches its local transaction list backward and stops at the first one that matches this transaction’s client name and number. It is the old incarnation of the same client and the same transaction number. Since the server didn’t receive the ‘commit’ message from the client before it encounters the new incarnation, the old incarnation must have been aborted. Therefore, the server simply deletes the old incarnation from its transaction list. Then the server adds a new transaction record with a ‘start’ log entry to its list. Whether or not the transaction is a new one, the server will add an ‘operate’ log entry to this transaction’s record. In this log entry, the primitive function to be called and the arguments to be passed to it are recorded. The function is identified by an integer number instead of a pointer to the function because it might be referenced after the system crashes and reboots. The function number is determined by the transaction stub generator.

When a ‘prepare’ message is received, the server tags the transaction as ‘ready’, adds a ‘ready’ entry to the transaction’s log and writes the log to stable storage.

When an ‘abort’ message is received, the server checks if the transaction is ‘ready’ or not. If it is, the server adds an ‘abort’ entry to the transaction’s log and writes it to stable storage. Then the transaction record is deleted from the list.

When a ‘commit’ message is received, the server adds a ‘commit’ entry to the transaction’s log and writes it to stable storage. Then it actually calls the primitive functions which were called during the transaction and delayed. These calls are guaranteed to be made at most once even if the sever receives the commit message from the client for more than once or the transaction is redone when the server reboots. Therefore, the primitive functions provided by the user do not have to be idempotent. The server locates a function by the function number in the log, which is actually an index to an extern array of function pointers. This array is declared in tpc_server.h , which is generated by the transaction stub generator.

Both the client and the server are multi-threaded, i.e. there could be multiple transactions active at the same time on both sides, but the actual execution of the transactions is serial, i.e. at one time only one transaction’s primitive actions are executed and the other transaction could not start executing until the current transaction finishes.

A checkpoint daemon thread is running in the transaction server for ever. It periodically writes all the datum in main memory to the disk and puts a check point in the log. Its purpose is to reduce the amount of the work that has to be done when system reboots. A checkpoint could be written only when no transaction is being committed. When the system reboots, all the transactions committed before the last check point need not to be re-done.

Possible Points of System Crashing

If a server crashes before the transaction enters the first phase of commit, the client will decide whether to abort the transaction or re-try. If the client crashes before the transaction enters the first phase of commit, the transaction will be aborted automatically. After the transaction enters the two-phase commit, crashes of either the client or any one of the servers will cause more complicated problems. See Figure 3 for details.

Client Reboots

When it reboots, the transaction clerk reads from the log in stable storage all those transactions that have a complete ‘commit’ entry but no ‘finish’ entry. It rewrites the log with only these ‘impending’ transactions in it. The client must have crashed after all the servers had ‘prepared’ for those transactions, some servers even had received the ‘commit’ messages from the client, but before the transactions were finally finished. Therefore, the clerk keeps re-connecting to the servers in the transaction’s servers’ list and sending them ‘commit’ messages by RPCs until all RPCs return successfully.

Server Reboots

When it reboots, the transaction server first reads all the system datum from disk. Then it reads from the log in stable storage all those transactions that did not abort or did not commit before the last checkpoint. It rewrites the log with only these ‘impending’ transactions in it. Then it re-does those transactions that have a ‘commit’ log entry and deletes them from the transaction list. It keeps the other transactions in the list. Either the client or the server must have crashed after those transactions were ‘prepared’ but before they were finally committed. Therefore the server will be waiting for further messages from the clients about those transactions. When it receives a message about a new transaction which has the same client name and number as one of those ‘impending’ transactions, it knows that the client has aborted the old incarnation of the transaction but it lost the ‘abort’ message or the transaction was aborted automatically. Therefore the server deletes the old incarnation from its transaction list.

The transaction server also starts a checkpoint thread when it reboots.

Transaction Stub Generator

See Figure 2 for the definitions, declarations and calls of the transact_ , remote_transact_ and transact_remote_transact_ versions of the primitive functions and files generated by the transaction stub generator and ACME RPC stub generator.

The biggest trick made by the stub generator is that it takes the primitive function list as input and outputs in tpc_server.h the declaration of an array of function pointers which point to those primitive functions. It also writes in tpc_server.h the definitions a group of macros which are indexes to the array. In tpc_server.c, where the transact_ functions are defined, each transact_ function knows which macro it has and thus writes the correct index of the array to the log. When the transaction server executes the primitive actions or re-does them after rebooting, it can correctly locate the functions by the array indexes.

Further Development

1. Change the underlying network protocols from ACME to TCP/IP, and the supporting Client-Server protocol from ACME’s to UNIX sockets’ , since a practical application should be built on top of widely-accepted protocols.

2. Write my own RPC stub generator instead of using ACME’s . The new stub generator will be based on UNIX sockets and combined with the existing transaction stub generator so that the user will be able to write only one description file and run the stub generator only once.

3. Implement the stable storage by replicating the files.

4. Cooperate closely with a widely-accepted naming protocol instead of totally relying on the application-specific naming.

5. Implement more properties of transactions such as serializability

6. Optimize the performance.

User Manual

Steps in writing a user-client and a user-server

The user, who must be programming in client-server style and using ACME RPC, is required to take the following steps:

1. Write a description file( user.x ) for ACME RPC generator which contains all the functions on the server side that are to be called remotely by the client.

2. Write a function list( tpc_func_list) that contains all the functions that are to be delayed during a transaction. Run tpcgen and get the following files:

tpc_server.c tpc_server.h tpc_client.c

tpc_client.h userServer.x

3. Run ACME's stubgen on userServer.x .

4. In the source code for user client, a typical transaction is as following:

{

ClTransaction * cltr;

cltr = startClTransaction();

error = transact_remote_transact_myFunc1(…cltr…);

if(error){

abortClTransaction(cltr);

break; }

error = transact_remote_transact_myFunc2(…cltr…);

… …

endClTransaction(cltr);

}

The transact_remote_transact_ functions are defined in tpc_client.c and declared in tpc_client.h . The user should include tpc_client.h in his source code for his client, compile tpc_client.c and link tpc_client.o and lib_trcl.a to his client program.

5. The user is also required to define the following function in his client source code:

void myClientName(char name[256]);

This function will be called by the transaction clerk library and should return the current client name.

6. The user is required to define the following functions in his server source code:

void myServerName(char name[256]);

void userOutputToDisk(void);

void userInputFromDisk(void);

These functions will be called by the transaction server library. myServerName should return the current server name, userOutputToDisk should write all the datum currently in main memory to the disk for backup and userInputFromDisk should read the datum from the disk to main memory.

See Figure 2 for making the whole system.

Data Structures

See cl.h and tpc_client.h for client side and sv.h and tpc_server.h for server side.

Files

• Files provided by the user

user.x tpc_func_list

Each line in tpc_func_list has the following format:

<Function Name> <Argument Size>

For details, see ‘Primitive Functions’ below.

• Files generated by the transaction stub generator

tpc_client.h tpc_client.c tpc_server.h tpc_server.c

userServer.x

• Files generated by the RPC stub generator

userServer_rpc.h userServer_serv.c userServer_clnt.c

Function Definitions and Calls

• Primitive functions

The executions of these functions will be delayed till the transaction in which these functions are called commits. Some of the them will also be re-done when system reboots. They are actually called in the transaction server library. The actions related to them must be atomic. All these functions must have the same format:

void <PFN>(char arg[ArgSize]);

PFN stands for Primitive Function Name. ArgSize is a constant that indicates the size of the argument in bytes. These functions must be described in the file tpc_func_list . All those operations that will modify the system data should be written in primitive functions.

• transact_ functions

These functions are generated by tpcgen for the primitive functions. They are defined in tpc_server.c and declared in tpc_server.h. They are also described in userServer.x so that a remote version could be generated for them by ACME RPC stub generator. They are called by the user server thread in response to an RPC request. They have the following format:

void transact_<PFN>(char name[256], int * num, char arg[ArgSize]);

The pair (name, *num) is used to uniquely identify a transaction on the server side. ArgSize is the size of the argument for primitive function <PFN> in bytes. The other arguments have the same meanings as previously defined. The description of these functions are automatically appended to userServer.x by tpcgen .

• remote_transact_ functions

These functions are generated by ACME RPC stub generator for the above transact_ functions, therefore they are defined in userServer_clnt.c and declared in userServer_rpc.h . They are called by the transaction clerk library in response to the user client’s calls for transact_remote_transact_ functions ( to be discussed below) . They have the following format:

int remote_transact_<PFN>(AcsConnection, char name[256], int * num, char arg[8]);

AcsConnection is the ACME connection to pass the RPC. The other arguments have the same meanings as previously defined.

• transact_remote_transact_ functions

These functions are generated by tpcgen for the remote_transact_ functions. They are defined in tpc_client.c and declared in tpc_client.h . They are called by the user client. They have the following format:

int transact_remote_transact_<PFN>(AcsConnection, char svName[256], ClTransaction * cltr, char arg[ArgSize]);

AcsConnection is the ACME connection between the user client and the user server which has the data this action is going to operate on. svName is the name of the user server, which acts as a static identification for this server, i.e. the transaction clerk should be able to establish a new connection with the server by calling anlConnect with svName as the argument when system reboots or the AcsConnection passed by the user client seems dead. cltr is the pointer to the transaction record in which this action is taken. The other arguments have the same meanings as previously defined.

• Functions provided by user client

void myClientName(char name[256]);

This function is called by the transaction clerk library, the returned name will be passed to the transaction server together with the local transaction number to uniquely identify the transaction. The name will also be used to constitute related file names.

• Functions provided by user server in addition to primitive functions

void myServerName(char name[256]);

This function is called by the transaction server library, the returned name will be used to constitute related files names.

void userOutputToDisk(void);

This function is called by the transaction server’s checkpoint thread. The checkpoint thread periodically calls userOutputToDisk to write all the datum in main memory to the disk and puts a check point in the log. When the system reboots, all the transactions committed before the last check point need not to be re-done.

void userInputFromDisk(void);

This function is called by the transaction server when initializing. It reads what userOutputToDisk wrote to the disk back to the main memory.

• Functions provided by transaction clerk

These functions are called by user client.

ClTransaction * startClTransaction(void);

This function starts a new transaction at the client side and returns the pointer to the transaction record.

int endClTransaction(ClTransaction * cltr);

This function terminates and tries to commit the transaction pointed to by cltr on all involved servers. If the transaction is committed successfully on all servers, the function returns 1. Otherwise, it aborts the transaction on all servers and returns 0.

void abortClTransaction(ClTransaction * cltr);

This function aborts the transaction pointed to by cltr on all involved servers.

• Functions provided by transaction server

These functions are called by user server thread in response to RPC requests.

void abortSvTransaction(char name[256],int * num);

This function aborts the transaction identified by the pair (name, *num) on the local server.

void prepareSvTransaction(char name[256], int * num);

This function adds a ‘ready’ entry to the log for the transaction identified by the pair (name, *num) and writes the log to stable storage.

void commitSvTransaction(char name[256],int * num);

This function adds a ‘commit’ entry to the log for the transaction identified by the pair (name, *num) and writes it to stable storage. Then it actually calls the primitive functions which were called during the transaction and delayed.

Figure 1. System Structure

Figure 2. Files and Funcitons

Figure 3. Possible points of system crashing

Client Crashes …

Server Crashes …