Serializing & deserializing Thrift data in Node.js

 ⋅ 3 min read

thrift-serde-nodejs

Apache Thrift allows writing an interoperable type-safe software stack. It comes with a code generation system that has its own definition language that can be converted to code across many programming languages.

As an example, you can write a User data structure that looks like this in Thrift and it can be used to auto-generate code in any language of your choice.

#User.thrift
namespace java com.httgp.models.thrift

struct User {
   1: required i16 id
   2: required string name
   3: optional string nickname
}

The auto-generated TypeScript definition for this Thrift model looks like this —

// User.ts
export interface IUserArgs {
    id: number;
    name: string;
    nickname?: string;
}

export class User {
    public id: number;
    public name: string;
    public nickname?: string;
    constructor(args: IUserArgs) {
        if (args != null && args.id != null) {
            this.id = args.id;
        } else {
            throw new thrift.Thrift.TProtocolException(thrift.Thrift.TProtocolExceptionType.UNKNOWN, "Required field[id] is unset!");
        }
        if (args != null && args.name != null) {
            this.name = args.name;
        } else {
            throw new thrift.Thrift.TProtocolException(thrift.Thrift.TProtocolExceptionType.UNKNOWN, "Required field[name] is unset!");
        }
        if (args != null && args.nickname != null) {
            this.nickname = args.nickname;
        }
    }
    // ...
}

The real benefit of using Thrift becomes obvious when you try to pass this data on to another service that is written in a different language. In this post, I will talk about how to do Thrift serde in Node.js and some common patterns.

Dependencies

  1. thrift for serialization & deserialization.

  2. node-int64 to handle 64-bit Ints.

  3. Optionally, @creditkarma/thrift-typescript for generating TypeScript definitions from your .thrift files. You can simply run —

thrift-typescript --outDir definitions User.thrift

Reading & writing Thrift data in Node.js

The Thrift documentation is quite sparse when it comes to their language-specific implementations, and after a lot of trial and error, here's how I managed to deserialize Thrift data in Node.js.

If you're consuming Thrift-serialized data (say, from a Kafka topic), the data is probably available in Node.js as a Buffer. The deserializeThrift method shows how to deserialize it —

import { TFramedTransport, TBinaryProtocol } from 'thrift';
import { User } from './definitions/User'; // Generated using @creditkarma/thrift-typescript

/**
 * Serializes native data of given model into Thrift.
 * @param data Data to serialize.
 * @param thriftModel Thrift model.
 */
function serializeThrift(data: object, thriftModel: any): any {
    const buffer = Buffer.from(JSON.stringify(data));
    const tTransport = new TFramedTransport(buffer);
    const tProtocol = new TBinaryProtocol(tTransport);
    const serializedData = data.write(binaryProt);
    return serializedData;
}

/**
 * Deserializes Thrift data with given model.
 * @param data Thrift data.
 * @param thriftModel Thrift model.
 */
function deserializeThrift(data: Buffer, thriftModel: any): any {
    const tTransport = new TFramedTransport(data);
    const tProtocol = new TBinaryProtocol(tTransport);
    const deserializedData = thriftModel.read(tProtocol);
    return deserializedData;
}
Deserialization
// rawData = getFromExternalDataSource(...);
const userObject = <User>deserializeThrift(rawData, User);
console.log(userObject);
// { id: 1, name: 'Ganesh', nickname: 'GP' }
Serialization
const userData: User = { id: 1, name: 'Ganesh', nickname: 'GP' };
const userDataAsThrift = serializeThrift(userData, User);
// Now you can write userDataAsThrift to your output sink (like a Kafka topic).

Handling Int64 values in JSON

While other languages have 64-bit Ints, JavaScript's Number supports only IEEE 754 double-precision floats, which are limited to 53 bits. The node-int64 package helps in handling them seamlessly by returning a custom Int64 object. However, if you wish to convert the Thrift-deserialized JSON into anything else, you'll need to manually handle Int64.

Fortunately, JSON.stringify() takes a "replacer" parameter that you can use to modify its default behaviour.

/**
 * Custom JSON stringify replacer.
 *
 * Converts `Int64` to `Number`. Returns same value if it isn't `Int64`.
 * NOTE: Won't be precise for VERY large numbers.
 */
function customStringifier(key: string, value: any): Number | any {
    if (value instanceof Int64) {
        return value.toNumber();
    } else {
        return value;
    }
}

// Convert deserialized object to a String —
JSON.stringify(deserializedObject, customStringifier);

Node.js
Code
Share