hermanjunge

These are the steps if you want to dump some text file into a redis database using node.js.

In most languages, the algorithm is straightforward. However there are some asynchronous considerations that are to be taken.

The repository is here. The idea is: You have a list in a text file. An example can be a female names list (borrowed from antirez’s blog), you want to introduce these names as key/value pairs, in this case, I wanted them to be ‘female:{name}/{name}’ . I divided the problem in two parts:

• Extracting the names from the file into the memory.
• Dump the memory contents into key/value pairs.

To solve the former problem, we must learn how a readStream object works: First this object is created with the method createReadStream() from the core module fs in node.js

var fileNamePath = './female-names.txt';
var readStream = fs.createReadStream(fileNamePath);

Then you must subscribe to the event ‘data’, this, is the emission of a buffer with the raw information to process

readStream.on('data', function(data) {
  var lines = lib.data2Array(data);
  lib.array2Pairs(lines, dbPrefix, client, function(err, res){
    if (err) { console.log(err); }
    });
});

As you can see, the event ‘data’ is subscribed by the anonymous function(data), which does all the processing needed by the application, specifically, the code line

var lines = lib.data2Array(data);

Calls the library function (which is in the file load2RedisTextLib.js) data2Array() Converting the stream of bytes into an array of names. Also, the logic of this function ignores the lines leading by a ‘#’, being those comments and the blank lines.

This function is synchronous and returns the array of lines

var data2Array = function(data) {

    var lines = [];
    var newline = '\n'.charCodeAt(0);
    var j = 0;
    var commentFlag = false;

    for(var i = 0; i < data.length; i++) {        
                
        if(data[i] == newline) {
            if(lines[j] != null) {
                if(!commentFlag && lines[j] != '') {
                     j++;
                }
                commentFlag = false;
            }         
        }
        else {
            if(lines[j] == null) {
                lines[j] = '';
            }
            var character = data.toString('utf8')[i];
            if(character == '#') commentFlag = true;
            if(!commentFlag) {
                lines[j] += character;
            } 
        }
    }

    return lines;
}

With our array of lines, our next task is defined by

 lib.array2Pairs(lines, dbPrefix, client, function(err, res){
    if (err) { console.log(err); }
});

Being the variables lines, the array of lines obtained in the former process, dbPrefix, the variable indicating the prefix in the Database (‘female:’), client, an instance of the redis client (using the module node-redis), and finally the callback of the method, which will alert us of any error we find.

The code for the function array2Pairs is 

var array2Pairs = function(lineArray, dbPrefix, client, callback) {

    lineArray.forEach(function(line) {
        client.set( dbPrefix + ':' + line, line, 
          function(err,result){
            if(err) {
                callback(err);
            }
          });
    });
}

And finally, we close the redis client subscribing to the end event of the readable stream

readStream.on('end', function() {
    client.quit();
    console.log('end');
});

If we want to check whether the key/value pairs were saved, we can do in the redis-cli program the (non-elegant and expensive) command

redis 127.0.0.1:6379> KEYS female:*

which will return the result of our endeavor:

And these were the basics of the operation of dumping the contents of a file intro redis.

- hermanjunge