Serve JSON Web Services with RPG and YAJL

aka — Access data running on other computers with REST and JSON

This is a resurrection of an old article which was lost from the internet when iPro Developer died. Luckily, I found an old PDF printout and dutifully copy/pasted all the text. Thanks to Scott Klement for this excellent article from Jul 2, 2013

Putting Web Services to Work on IBM i – On Demand

Every organization has more than one computer. So it’s only natural that companies need a way for programs on one computer to call programs running on others. The server running the HR software, for example, might need to call the server running the sales software to determine how much commission to pay the company’s salespeople in a given month.

Over the years, the IT industry has developed many ways for programs to call other programs, but the best way I’ve found is web services. In this article, I’ll discuss how to write a Representational State Transfer (REST) web service in RPG that returns data in JavaScript Object Notation format. But that’s where the similarities end. JSON is lighter weight—it expresses the same data in fewer characters than XML. In addition, JSON is always coded in UTF-8 (it doesn’t support other encodings), making it simpler to deal with—and you don’t have to worry about it sometimes being in ASCII or other flavors of Unicode and then having to deal with all those possibilities in your program, because you know it’s always UTF-8.

My favorite thing about JSON is that it’s a subset of JavaScript, in which it’s used as a way to initialize variables. As an analogy, RPG provides several ways to initialize data. To initialize stand-alone variables, RPG has D-spec’s INZ keyword. To initialize arrays or tables, it has compile-time data (the data you put beneath the ** line at the bottom of your program.) In JavaScript, however, you use JSON to initialize data, and that’s why I use JSON for web services that communicate with JavaScript programs—because JSON is easy to use from JavaScript, and it performs better than XML.

Yet Another JSON Library

YAJL is an open-source JSON generator and parser that was created by Lloyd Hilaiel, who works for Mozilla and, therefore, is—without a doubt—very familiar with browser technologies like JSON. YAJL provides two ways to parse JSON data, as well as the JSON generator that I demonstrate in this article. Why use YAJL instead of another JSON tool? Well, last year, a colleague and I at Profound Logic looked for the best JSON parser to use in our software. We needed something that would be fast, because our software is known for its high performance, and were worried that JSON parsing might slow it down. We investigated a dozen different JSON tools and found that YAJL was by far the most efficient. It was double the speed of the next fastest tool!

YAJL is written in C and designed to be called from C programs. However, I wrote an RPG front end to make it easy to use YAJL from RPG programs. You can get a copy of YAJL, my RPG front end, and the sample code demonstrated in this article from my website at scottklement.com/yajl.

Build a Web Service

To provide an example for this article, I’ve written a web service in RPG that retrieves stock status. You can request the status of an individual item in stock (or if no item is specified, the web service will return all items). The basic idea is that the caller provides a Uniform Resource Identifier (URI) in the format https://your-system.com:8500/stockqty/itemno, where “your-system.com:8500” is replaced by the IP address or domain name of the computer running the service and the port number of the HTTP server. The itemno portion of the URL is optional. If supplied, the service will provide only the stock status for that item; if it’s not provided, the service will provide stock status for all items.

monitor;
  uri = getURI();
  itempos = %scan('/stockqty/': uri) + %len('/stockqty/');
  itemid = %subst(uri: itempos);
on-error;
  itemid = *blanks;
endmon;

readData(itemid: data);
createJSON(data: jsonBuf);
sendReply(data.success: jsonBuf);

return;

The program starts by retrieving the URI that was used to call the web service:

uri = getURI();

Because the host name and port might be different in different environments, it skips past them by scanning for the string “stockqty” in the URI:

itempos = %scan('/stockqty/': uri) + %len('/stockqty/');

and then taking a substring of the data that comes after it:

itemid = %subst(uri: itempos);

If there’s no data after it, or if an error occurs, the item number is set to blanks:

itemid = *blanks;

The remainder of the mainline calls three subprocedures: readData, createJSON, and sendReply:

readData(itemid: data);
createJSON(data: jsonBuf);
sendReply(data.success: jsonBuf);

Define Data Structures as Templates

The readData procedure takes the item number as input and reads a physical file to get the stock status data. This article assumes you know how to read a file in RPG, so I didn’t provide the code for the readData subprocedure. Suffice it to say that the output is the data structure in Figure 2, which shows the data structure’s layout.

Data structure template that loads from the physical file

D item_t ds qualified
D template
D prid 10a
D prname 30a
D pprice 11p 2
D pstock 5p 0

D data_t ds qualified
D template
D success 1n
D errmsg 80a
D count 10i 0
D item likeds(item_t) dim(999) 
D data ds likeds(data_t)

It consists of a main data structure template named “data_t” that contains subfields for tracking whether the data was loaded successfully (“success”), the error message if it’s not loaded successfully (“errmsg”), the number of items loaded (“count”), and an array of up to 999 items.

D data_t ds qualified
D template
D success 1n
D errmsg 80a
D count 10i 0
D item likeds(item_t) dim(999)

The items array uses the LIKEDS keyword to refer to the item_t data structure (Callout A in Figure 2), which contains the product ID, name, price, and stock quantity of each item returned.

D item_t ds qualified
D template
D prid 10a
D prname 30a
D pprice 11p 2
D pstock 5p 0

I like to define data structures as templates that I can clone with the LIKEDS keyword because I can use the template structures on prototypes for subprocedures. I also used the template to create a data structure named “data” (Callout C in Figure 2) that I can use as a parameter when I call the subprocedures:

D data ds likeds(data_t)

Create JSON From the Data Structure with YAJL

Now that the data is loaded into the data structure, the next step is to create JSON data in the same format as the data structure. This way, my web service consumer (written in JavaScript) can use the data the same way that the RPG program uses it. Before I explain how to call YAJL, look at Figure 3, which shows an example of the JSON data that will be sent.

Example of JSON data

{
"success": true, "errmsg": "", "count": 3, "item": [
{
"itemno": "ABC AC904",
"name": "ABC ACCESSORY BELTS", "price": 7.29,
"stock": 50
},
{
"itemno": "ABC AV889",
"name": "ABC AVALANCHE PROBE", "price": .35,
"stock": 50
},
{
"itemno": "ABC BE839", "name": "ABC BELAY SEAT", "price": 15.61,
"stock": 50
}
]
}

JSON is structured so that the curly brackets begin a new object. It’s worth noting that an “object” in JSON is equivalent to a “data structure” in RPG. Within each object in the JSON data, you find field names in quotes followed by a colon, followed by the value of that field. Each field is separated from the next field by a comma. In Figure 3, you find the first three subfields of the object:

"success": true, "errmsg": "", "count": 3,

The success subfield is a Boolean subfield, meaning that it can be set to true or false. In this example, it’s set to true to indicate that the web service was successful. The “errmsg” subfield determines error messages (in this case, it’s an empty string because there was no error). The count subfield is set to 3 to indicate that there are three items returned.

You can see the following returned items:

{
"itemno": "ABC AC904",
"name": "ABC ACCESSORY BELTS", "price": 7.29,
"stock": 50
},

{
"itemno": "ABC AV889",
"name": "ABC AVALANCHE PROBE", "price": .35,
"stock": 50
},

{
"itemno": "ABC BE839", "name": "ABC BELAY SEAT", "price": 15.61,
"stock": 50
}

The square brackets before and after each item indicate an array of repeating data. Each item in the array is separated from the next by a comma. Note that each item begins and ends with a curly bracket; this is because the item array contains objects, and each object has four subfields. In other words, it’s the same format as the RPG data structure. The main data structure contains an array, and within that array are data structures that contain four subfields. The fact that these subfields are output in JSON format means the JavaScript routine that calls this web service has data in the same format as the RPG program.

Now that you know JSON’s format, look at the RPG code that generates it. Figure 4 shows the createJSON subprocedure, which uses the routines in YAJL to create the JSON data.

RPG subprocedure that generates JSON data with YAJL

yajl_genOpen(*on);

yajl_beginObj(); 
yajl_addBool('success': *on ); 
yajl_addChar('errmsg': %trimr(data.errMsg)); 
yajl_addNum('count': %char(data.count) );

yajl_beginArray('item');
for x = 1 to data.count;
yajl_beginObj();
yajl_addChar('itemno': %trimr(data.item(x).prid) ); 
yajl_addChar('name' : %trimr(data.item(x).prname)); 
yajl_addNum( 'price' : %char( data.item(x).pprice)); 
yajl_addNum( 'stock' : %char( data.item(x).pstock));
yajl_endObj();
endfor;
yajl_endArray();

yajl_endObj();

%len(jsonData) = JSON_BUF_SIZE;
rc = yajl_copyBuf( 1208
: %addr(jsonData: *data)
: JSON_BUF_SIZE
: len );
%len(jsonData) = len;

yajl_genClose();
return rc;

The first step is to call the yajl_genOpen subprocedure, which initializes YAJL’s JSON generator:

yajl_genOpen(*on);

The only parameter you need to pass to yajl_genOpen is an indicator that should be *ON if you want your JSON data formatted to make it easy for a human being to read, or *OFF if you want the data as compact as possible. Personally, I set this indicator to *ON when I’m debugging my program because it makes the output much easier to read. Once I’m satisfied that my program is working properly, I turn it off, which squishes the JSON data all onto one line and makes it very compact. Because the compact data uses fewer bytes, it transmits faster over the network, and therefore my application runs faster.

YAJL’s JSON generator contains routines that start a new object (JSON’s equivalent of a data structure); start a new array; and insert fields that are in character, numeric, or Boolean (true/false) format. The basic idea is that once you begin an object, any fields you add are considered a part of that object. Likewise, once you begin an array, any fields you add are added to the array. There are also routines that end an object or array—always ending the most recent object you add.

Because the returned data is in one data structure, and I want the JSON code to work the same way, I call yajl_beginObj to begin an object:

yajl_beginObj();

This outputs the curly bracket that begins an object. This is followed by calls to yajl_addBool, yajl_addChar, and yajl_addNum, which output the success, errmsg, and count subfields (along with their values) to the JSON document:

yajl_addBool('success': *on );
yajl_addChar('errmsg': %trimr(data.errMsg));
yajl_addNum('count': %char(data.count) );

Because I’ve begun an object but not yet ended it, these fields are added to that object. It’s not necessary to escape any special characters in my RPG data because YAJL takes care of that for me. It also converts the data to UTF-8. Each time I begin a new object or array, I like to indent the subfields that belong to it. I think this makes it easier to read the code, because you can immediately see that the subfields are added to an array.

The next step is to insert the array of items within the same object. This is done by calling the yajl_beginArray routine to start the array and output the square bracket to the JSON document:

yajl_beginArray('item');

In RPG, I use a for loop to add each item object to the array:

for x = 1 to data.count;
yajl_beginObj();
yajl_addChar('itemno': %trimr(data.item(x).prid) ); 
yajl_addChar('name' : %trimr(data.item(x).prname)); 
yajl_addNum( 'price' : %char( data.item(x).pprice)); 
yajl_addNum( 'stock' : %char( data.item(x).pstock));
yajl_endObj();
endfor;

Because I haven’t yet ended the array, each new object I add is considered part of the array, and I use indenting to make that as clear as possible. Once the loop is complete and I’ve added all the items to the array, I end it by calling the yajl_endArray subprocedure, which inserts the closing square bracket and tells YAJL that the array is complete:

yajl_endArray();

Any additional fields I write won’t be part of the array. Because it’s possible to have multiple arrays in a single JSON document, yajl_endArray always ends the most recent array that was started. It’s also possible to nest arrays inside other arrays. If I’d wanted to do that, I’d have called yajl_startArray without ending the prior array, causing the second array to appear inside the first. In the final step, the createJSON routine calls yajl_endObj to end the object that contains the entire JSON response and inserts the closing curly bracket:

yajl_endObj();

I’ve finished building my JSON data, and now I need to load it into an RPG variable so I can send it back to the HTTP server, which sends it back to the web service consumer. To do that, I call a routine named yajl_copyBuf, which copies the data from YAJL’s internal JSON buffer to my RPG variable named jsonData. But, because jsonData is a VARYING field, and yajl_copyBuf works with raw buffers before I can copy the data, I need to set jsonData to its maximum length:

%len(jsonData) = JSON_BUF_SIZE;

I then call yajl_copyBuf to copy the data from YAJL’s internal data buffer to my RPG field:

rc = yajl_copyBuf( 1208
: %addr(jsonData: *data)
: JSON_BUF_SIZE
: len );

and set the length of jsonData to the proper length so that the VARYING field works correctly:

%len(jsonData) = len;

Finally, my JSON is in the jsonData field, and I can tell YAJL that I’m done with its JSON generator by calling yajl_genClose:

yajl_genClose();

The yajl_copyBuf routine that I demonstrated earlier accepts four parameters. They are:

1. The Coded Character Set Identifier (CCSID) of the output data. I can ask YAJL to convert the data to any CCSID I need. In this case, I’ve used CCSID 1208, which is UTF-8.
2. The address of the buffer where I want to receive data. This can be the address of an RPG alphanumeric variable (either VARYING or fixed length), or it can be a pointer to a buffer that I allocate in memory.
3. The size of the buffer in parameter two.
4. A variable to receive the length of the data (in bytes) that YAJL stores in my buffer.

Send a Reply to the Web Service Consumer

The sendReply subprocedure sends JSON data to the HTTP server so that it can be sent to the web service consumer. This shows a snippet from this subprocedure.

if data.success = *off;
headers = 'status: 500 Internal Server Error' + CRLF +
'Content-type: application/json; charset=utf-8' + CRLF
CRLF;
else;
headers = 'status: 200 OK' + CRLF +
'Content-type: application/json; charset=utf-8' + CRLF + CRLF;
endif;

QtmhWrStout(headers: %len(headers): err); 
QtmhWrStout(jsonData: %len(jsonData): err);

At the following lines in Figure 5, the code generates HTTP headers that tell the HTTP server the type of data I’m sending (application/json identifies the data as JSON):

if data.success = *off;
headers = 'status: 500 Internal Server Error' + CRLF
'Content-type: application/json; charset=utf-8' + CRLF
CRLF;
else;
headers = 'status: 200 OK' + CRLF
'Content-type: application/json; charset=utf-8' + CRLF
CRLF;
endif;

The program calls the IBM-supplied QtmhWrStout API to send both the headers and the JSON data to the IBM HTTP server (Powered by Apache), which sends the data to the web service consumer:

QtmhWrStout(headers: %len(headers): err); 
QtmhWrStout(jsonData: %len(jsonData): err);

Install the Web Service

Before the web service can be called, you need to tell the IBM HTTP server about it. I chose to create a library named SKWEBSRV, in which I put the web services that I want to make available. Next, I go into the HTTP Admin server (running on port 2001) and choose Internet Configurations, IBM Web Administration, All Servers to select an HTTP server that I’d like to add my web service to.

If you don’t already have an HTTP server, you can create: Click Create HTTP Server and an easy-to-use wizard will guide you through the process. Don’t click the Create Web Services Server, because it uses the IBM Integrated Web Services tool for creating SOAP web services. For this article, I’m using an ordinary HTTP server.

In the configuration for the HTTP server you’d like to add the STOCKQTY web service to, scroll down the left navigation pane and choose Edit Configuration File to add the keywords shown in Figure 6.

Apache directives to enable the web service

ScriptAlias /stockqty /qsys.lib/skwebsrv.lib/stockqty.pgm
SetEnv QIBM_CGI_LIBRARY_LIST "SKLEMENT;SKTEST;SKTRAIN;QGPL;QTEMP" Order Allow,Deny Allow from all

The ScriptAlias keyword tells the HTTP server that any URL beginning with “stockqty” will call the program named STOCKQTY in library SKWEBSRV:

ScriptAlias /stockqty /qsys.lib/skwebsrv.lib/stockqty.pgm

The group of directives sets up the library list for the SKWEBSRV library and grants the HTTP server permission to let anyone call programs in this library:

SetEnv QIBM_CGI_LIBRARY_LIST "SKLEMENT;SKTEST;SKTRAIN;QGPL;QTEMP" Order Allow,Deny Allow from all

Call the Web Service

To call the web service, I wrote a web service consumer. It consists of a simple HTML page containing JavaScript code that uses Asynchronous JavaScript and XML (Ajax) to call the web service.

Handling the JSON response in JavaScript

var data = eval("(" + req.responseText + ")");
var table = '<table border="1">'
+ "<tr>"
+ "<td>Item No</td>"
+ "<td>Item Name</td>"
+ "<td>Price</td>"
+ "<td>Stock Qty</td>"
+ "</tr>";
for (var x=0; x<data.count; x++) {
table += "<tr>"
+ "<td>" + data.item[x].itemno + "</td>"
+ "<td>" + data.item[x].name  + "</td>"
+ "<td>" + data.item[x].price + "</td>"
+ "<td>" + data.item[x].stock + "</td>"
+ "</tr>";
}
table += "</table>";
var result = document.getElementById("result");
result.innerHTML = table;

This code receives the JSON response from the HTTP server and converts it to a JavaScript variable by invoking the JavaScript compiler through the eval() function:

var data = eval("(" + req.responseText + ")");

As you can see, it’s extremely easy to convert JSON data to a JavaScript variable. Next, the code loops through the data that was returned and uses it to build an HTML table:

var table = '<table border="1">'
+ "<tr>"
+ "<td>Item No</td>"
+ "<td>Item Name</td>"
+ "<td>Price</td>"
+ "<td>Stock Qty</td>"
+ "</tr>";
for (var x=0; x<data.count; x++) {
table += "<tr>"
+ "<td>" + data.item[x].itemno + "</td>"
+ "<td>" + data.item[x].name  + "</td>"
+ "<td>" + data.item[x].price + "</td>"
+ "<td>" + data.item[x].stock + "</td>"
+ "</tr>";
}
table += "</table>";

Give It a Try

Now that you know how to use your RPG programming skills to write a REST web service that returns data in JSON format, the best thing for you to do is install and try it. You can walk through the RPG code in debug mode and see for yourself how the program works.