At a glance, that can look bad. Any potential employer is going to think "Oh this guy has just been chasing money and/or a promotion!". Delve deeper, and you'll know my story, and I'm sure countless others', is mired with toxic workplaces and awful bosses.

My little journey

I got my first taste of computers when I was around 10, and I started building simple quiz apps in Visual Basic using a friend's computer (we couldn't afford one). When I went to high school, I finally got my own computer and started building tiny apps with C#; I was also featured on LifeHacker. I built websites for college events using Flash and PHP, and for my final year project, I built a robot that could be driven via commands from a serial port. I finally landed a job straight out of college at one of the most popular software companies (at the time) in the country.

I spent 3 years at my first job. I was young, and I was learning some cool stuff — I was part of a research group that worked closely with IBM to come up with a plan to port an incredibly popular core banking product's database from Oracle to DB/2. But working for a large multinational corporation, it became apparent that I wouldn't be able to create impact in many years, and I was being criminally underpaid. I had been teaching myself Python on the side (I'd written a bunch of scripts to automate parsing of DB/2 logs, cutting down the time my team was spending on that effort by almost 90%), and I was ready to go look for Python jobs.

And I was a Python/Django/jQuery developer for a while. Node.js was starting to become a thing, and I became a Node.js developer and found a job as one. I was also starting to work on Angular. Soon I ended up at a fantastic place that would become my engineering foundation and my baptism-by-fire; I was working on Scala, Kafka, Mesos, Ansible, Terraform & Packer at scale. I was working on unbelievably cool stuff like working around bot mitigation techniques that were sold by Distil and Akamai. This was also where I was finally in a room full of people smarter than me. The company, however, ran out of runway and was soon acquired by somebody else, and the transition was incredibly painful. I left the job really hurt, and hastily picked another one as a rebound.

Like most rebound relationships, it did not last. It was an incredibly toxic workplace — one of the founders was an insufferable bully who publicly berated people, pay disparity was huge, personal boundaries were disrespected and delivery expectations were so stressful that at some point I started literally losing my hair. I quit that job in 7 months.

I found another job, and by this time, experience and wisdom informed what I specifically wanted. I wanted a job where —

1. Everybody built stuff in the open.
2. People would be treated with kindness and compassion.
3. I could build inclusive and diverse teams filled with smart people that were paid equitably.
4. I could attempt to fix the fundamentally-broken hiring process.
5. I could help solve interesting problems.

Armed with this list of goals, I started work in earnest at my last job. While the first year was spent entirely as the sole contributor building the platform frontend, it was time to grow the team, and I'd soon find out why I'd leave this job too; management reneged on promises of equitable pay, information trickled-down from the top, new product managers were solving problems that didn't exist, and I was frequently left out of hiring decisions and other important meetings.

Who's Fault Is It Anyway?

When trying to analyze what I could've done to solve these problems, I'm constantly reminded of what an old boss told me —

"You're just running away from your problems. If something isn't going your way, it's your responsibility to fix it!"

There is some truth to this — I was technically running away from those problems. But where I disagree is that was somehow my problem. In that particular case, I decided to leave because I was brought in to modernize and turn a maintenance shop into a development shop. This meant I had to bring in new processes, mentor existing employees and try to shape them into full-fledged engineers, and get a hiring budget to attract good talent. I wasn't able to do any of this because the company's overseas branch didn't like us doing any new development, the team was very resistant to all the new changes and I couldn't hire anyone I liked. Throughout all of this, I wasn't getting any support from the people who hired me either, so I was just stuck; until, of course, I decided to leave.

People don't leave bad jobs, they leave bad managers.

Should one try to solve problems at their organization? Yes, of course. Is my team falling behind on schedule? I'll try to understand why we're late, identify bottlenecks and try to fix those. Is someone on my team not being productive? I'll talk to them, see if they need a break or therapy, and I'll help them get that. But I can do all of this only if I am supported, and know that the company cares about this too. Maybe it is my ADHD brain's acute sense of fairness, but I need my employer to also care about some of the things I care about — diversity, inclusivity, fair & considerate hiring processes, kindness and compassion not just for customers but for employees too, mental health wellness and equitable pay. If they're only focussed on turning a profit (like this one job I had where a core value, enshrined in all their decor & company swag, was "Results over reason"), I'd be incredibly miserable there; you would too.

End of the rope

At every job, I've fought very hard to fix those things. I've been explicit about how I want certain things to be done. I've pleaded, begged and persuaded bosses to treat and pay people better. I've battled long and hard for better working hours for my teams, and I've learnt new ways of talking to non-technical folks to help them understand and appreciate what software engineers do. I've had honest and thoughtful responses for anyone hiring me that had concerns about me being a "flight risk". All this said, I've left most of those jobs when I no longer could pay the cost of trying to fix everything.

Stage 5: Acceptance?

My partner keeps telling me that the only way I could find happiness at a job is if I ran my own company and did everything the way I want it to be.

But I've seen great employers that know how to treat their folks really well. I've been a part of at least 1 team where I've experienced a version of happiness that is close to my ideal. And I refuse to believe that I can't recreate that again.

That said, I do realize that every organization, at the end of the day, exists to generate wealth for its founders, management and shareholders. Will a company exit by selling to someone that pays lesser to the shareholders while elevating the employees? Most probably not. I'm trying to come to grips with this hard reality by detaching more from my company as an identity and by looking for happiness outside of my job. I am also advising all of my teammates to do the same — I encourage them to pursue fulfilling hobbies, to exercise and eat right, to spend more time with their families, friends & pets and to volunteer.

How to not repeat my mistakes and instead make your own

So how does one avoid the traps of a bad job in the first place? I've learnt that there is no hard science to this, but I now do a few things —

1. When approaching a new job, I do a ton of research into the company and its founders. What is their story? What does their social media look like? Are there any obvious red flags? Do they engage in the things I care about?
2. I read their job listings. Do they talk about the realties of the job? (Mighty's hiring document is a great example where they're refreshingly honest about the importance of raw hours). Do they have a near-future roadmap of the role?
3. When I start talking to the founders, I ask pointed questions about their mission and their long-term goals, their exit strategy, where they see my adding value immediately, in what ways the founders (if there's more than 1) are different and same, what their team could be doing better right now, their CSR plan, have they had to fire someone (and if yes, why), how they measure employee happiness (and how they define it), how much support they provide for new parents and how they plan to do appraisals. I'm always making jokes, so I see if they get my sense of humour.
4. When interviewing, I also ask to meet the teams I'll be working with. This gives me an opportunity to understand more about the people themselves and to start building a relationship.
5. I am very honest about all the reasons I've quit past jobs, and I explicitly state the things I want and deeply care about. I've had founders nod their head along at this stage (because they want me onboard, so they'll agree to anything I was saying), so I'm extra careful here about probing deeper and making sure they're earnest and actually mean it.
6. If there are any red flags along the way, I try not to fall prey to the sunk cost fallacy ("Oh I've spent so much time in this process, I'd hate to start over again!") and still withdraw from the process.
7. I do my own due diligence — I talk to current and previous employees if possible, and ask specific questions about the company culture and the reasons for their departure.
8. I listen to my gut. If something feels off, I walk away.

A note on being able to afford being this way

I have to point out that this isn't financially viable in the long run, nor is it even possible in the first place if I wasn't already incredibly privileged. I would be remiss if I didn't acknowledge the things that have allowed me to take such principled decisions.

1. My parents were first-generation college (or even school) graduates, and they worked hard to give me a middle-class upbringing. I never had to worry about food, clothing, a roof over my head, school fees, saving for a sibling's wedding or taking care of my parents. Consider all of this, and I was incredibly privileged to have a stable household.
2. My partner is employed, and we don't have kids. Our DINK lifestyle has de-risked this a lot.
3. I'm confident in my employability skills, so I could quit and take breaks in between jobs without fretting over finding a new one.

One of the things that every Terraform developer will end up having to do at some point is conditionally bringing up on-demand or spot instances in their AWS infrastructure, and I'm going to show you how to do that.

The challenge

The main challenge in doing this is not the bring up itself, but propogating your EC2 tags down to the instance. This is because to bring up a spot instance, you'll need to use aws_spot_instance_request, which takes tags as an argument but applies them to the spot request itself and not the instance.

Settings tags on the spot instance

The solution involves using count to conditionally bring up the instance, creating an IAM policy that allows the instance to create tags and then running remote-exec to SSH into the instance to create its own tags.

Prerequisites

1. AWS AMI with AWS CLI installed.
2. Terraform >= 0.12.

variables.tf

variable "instance_type" {
  type        = string
  description = "Instance type for your AWS instance"
}

variable "ami_id" {
  type        = string
  description = "AMI ID for your AWS instance"
}

variable "ssh_key" {
  type        = string
  description = "EC2 SSH keypair for your AWS instance"
}

variable "instance_lifecycle" {
  type        = string
  description = "Lifecyle of your AWS instance. Example: ondemand, spot"
  default     = "spot"
}

ec2.tf

resource "aws_instance" "ec2_ondemand" {
  count                       = var.instance_lifecycle == "ondemand" ? 1 : 0
  ami                         = var.ami_id
  instance_type               = var.instance_type
 
  tags = {
    Name        = "my-instance"
    Lifecycle   = var.instance_lifecycle
  }
}

resource "aws_spot_instance_request" "ec2_spot" {
  count                       = var.instance_lifecycle == "spot" ? 1 : 0
  wait_for_fulfillment        = true
  spot_type                   = "one-time"
  ami                         = var.ami_id
  instance_type               = var.instance_type

  tags = {
    Name        = "my-instance"
    Lifecycle   = var.instance_lifecycle
  }

  # Workaround to make sure the spot request tags are propogated down to the instance itself.
  provisioner "remote-exec" {
    connection {
      user        = "ubuntu" # This might change for your OS of choice.
      host        = self.private_ip
      private_key = file("~/.ssh/${var.ssh_key}.pem")
    }

    inline = [
      join("", formatlist("aws ec2 create-tags --resources ${self.spot_instance_id} --tags Key=\"%s\",Value=\"%s\" --region=${var.region}; ", keys(self.tags), values(self.tags)))
    ]
  }
}

# If you're going to refer to the created instance anywhere else, you should now use `data.aws_instance.ec2_instance`
data "aws_instance" "ec2_instance" {
  depends_on = [aws_instance.ec2_ondemand, aws_spot_instance_request.ec2_spot]
  filter {
    name   = "tag:Name"
    values = ["my-instance"]
  }
  filter {
    name   = "tag:Lifecycle"
    values = [var.instance_lifecycle]
  }
}

iam.tf

resource "aws_iam_role" "iam_role" {
  name = "my-iam-role"

  lifecycle {
    create_before_destroy = true
  }

  force_detach_policies = true

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

}

resource "aws_iam_policy" "iam_role_policy_ec2tags" {
  name        = "my-instance-create-tags-policy"
  description = "Allow EC2 instances to create Tags for themselves"

  # The policy is for '*' rather than the own instance because EC2 doesn't allow us to filter by self.
  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Action": "ec2:CreateTags",
          "Effect": "Allow",
          "Sid": "AllowEC2InstanceCreateTags",
          "Resource": "*"
        }
    ]
}
EOF

}

resource "aws_iam_policy_attachment" "iam_role_policy_ec2tags" {
  name       = "iam_role_policy_ec2tags"
  roles      = [aws_iam_role.iam_role.name]
  policy_arn = aws_iam_policy.iam_role_policy_ec2tags.arn
  lifecycle {
    create_before_destroy = true
  }
}

What is a platform team?

Although most organizations tend to describe a platform team slightly differently, on a fundamental level, a platform team is the team that has centralized expertise and ownership. At the same time, it should allow a healthy amount of control and customization by product teams. The platform team is also broadly responsible for improving the organization's flexibility and developer productivity.

How to build a platform team

Identify roles & responsibilities

Answering some questions like what are the team's roles (which pieces of the software puzzle will they build/own/maintain) and responsibilities (what they're accountable for, what they can do to build a working relationship with other teams, how often they communicate with other stakeholders, etc.)

Write a mission and/or vision statement

The team's mission statement should help align the team on its core goals and build a sense of camaraderie and shared ethos.

Poach & grow talent internally

Often at an organization's growth phase, there are pockets of tribal knowledge and SMEs that are invaluable in understanding a system. They are the flesh-and-blood documentation of a system, and can be much more quicker at adding value to a platform team than a new hire.

Build trust

Many engineers are often resistant to change and care deeply about things they've built. Often times the platform team can come in and re-architecture or change things, leading to some friction. The platform team needs to build trust first before they can get buy-in from the development team; after all, they're the platform team's customers.

Lay down best practices

Unlike development teams, one of the stated goals of a platform team is very good documentation and detailed guides. This helps immensely in —

1. Boosting developer productivity.
2. Clearly communicating the team's vision.
3. A solid framework to iterate upon.

Documentation is the lifeblood of the team and it cannot be seen as a separate task in a sprint; it has to be given equal importance as (sometimes even greater than) code.

Take ownership and build accountability

As the platform team starts to take shape, taking voluntary ownership of either legacy systems or new systems that they've built is important. This frees up developers to focus on external customers' deliverables. Becoming accountable shows that the team is growing and maturing, and that they're a key part of the chain of trust.

Establish a communication API

Crisp & inclusive communication of what the team is working on, what it plans to achieve, the risks it perceives and helpful tips to developers (even as JIT) at a predictable cadence is very important to build more trust and effectively participate in larger efforts inside the organization. This can also double as free PR for the team!

Create a feedback loop

Every member of the team can appreciate —

1. An external frame of reference.
2. Validation for their ideas.
3. Gratitude from seeing their work positively impact a developer or a product.
4. Inputs that drive them to build cooler stuff.

A feedback loop manages to keep the platform team engaged with their customers (developers and product teams). It offers valuable insight into their core demographic, and helps not silo them away from the rest of the organization.

Publish data, metrics and internal SLAs

The team should regularly publish data on —

1. Development progress.
2. Technical backlog.
3. Issues fixed.
4. Performance improvements made.
5. Security threats mitigated.

A dashboard or a status page can help communicate the team's effectiveness to everyone at the organization. Additionally, at a certain level of maturity and stakeholder confidence, it is worthwhile setting up an internal SLA that holds the team to a higher standard.

Enable a self-serve platform

As the tools and the platform itself matures, they should evolve in the direction of automation. This becomes a force-multiplier for the team itself, making room for even faster iterations, bigger goals and a more stress-free environment for the platform engineer.

Challenges & possible solutions

While there's a lot of compelling arguments for building a platform team, there are also a few challenges that they can face.

Disillusionment & disenfranchisement

More times than often, the platform team is a disruptor. It has the wisdom of hindsight and knowledge of past mistakes to learn from, and it aims to fix a lot of things. This can often understandably lead to a sense of disillusionment (and perhaps a sense of betrayal) in some engineers that wrote legacy code.

This can be handled by bringing on those same engineers to the platform team either full-time or as loans, giving them an opportunity to fix their old mistakes (because a good engineer will always do that happily).

Fragmentation and more silos

A classic growing pain in any startup is fragmentation of expertise, and ever-so-slightly increasing over-specialisation. As more teams are formed with a more specific focus, knowledge tends to become hidden in pockets more and more.

To be honest, I personally don't have a good solution for this problem. Perhaps it is wiser to accept this as a part of growth and evolution. What can perhaps make it easier to gain wider acceptance is clearly set expectations upfront. Cross-functional knowledge sharing can be built with more documentation (perhaps tailored to team-specific personas) more forced collaboration (internal hackathons, regular presentations, etc.).

Loss in development velocity

High development velocity is highly favoured in the world of agile startups. Developers tend to view any additional process as bureaucracy and view them as a waste of time. The platform team has its own deadlines, but also has to efficiently manage the time that the development teams' output spends in its quality gates (compliance and regulatory requirements, specific Dev(Sec)Ops tool checks, etc.).

Striking a good balance between keeping the development teams moving forward but also making sure all best practices checks have passed is a skill that only be developed on-premises, and it is very finely tuned to the organization's culture and it's people.

Growing technical debt

Technical debt is the bane of developers' existence, and we all wish there was a magic wand we can wave to pay it all off. However, in the real world, the onus of clearing it often falls on the platform team. This can be a double-edged sword, as it can often be the draw for some engineers, but can be a deterrent for others.

This can be solved via regular internal hackathons based on debt. Perhaps additional incentives (not necessarily in monetary form) can be given to independent contributors and teams that clear their backlog. This can also be gamified — for example, a team can gift anybody X "brownie points" for clearing off some debt item; "brownie points" is a placeholder for an incentive of some kind.

Impedance mismatches in prioritization

Following best practices will not always be the top priority for a lot of development teams, while it is the other way around for a platform team. There will almost always be some impedance mismatch between the two teams because of this and every member has to be sensitive to this.

Project managers can help strike a balance here. The platform team can also groom champions or ambassadors that can rally around the team around compliance with the best practices checklist.

Trust-building

Building trust is of prime importance, doubly so for a new team. Also compounding this issue is that the security team has a tangentially different focus than the other teams, requiring way more effort in building trust and gaining equal footing at the organization.

This will probably be a slow process (especially if there are new hires on the team), but can be achieved with heavy focus on transparency, relying on data as truth, seeking buy-in from other teams and involving them in platform decisions and finally some good old dogged persistence.

Continuing with my series of Mesos posts, I wanted to show how to correctly configure log rotation for your Mesos agent sandboxes. The official documentation on Mesos logging leaves much to be desired in terms of real-world examples, so here's how I did it successfully.

Steps to set up log rotation

1. Set the --container-logger Agent flag to org_apache_mesos_LogrotateContainerLogger.
2. Set the --modules Agent flag to file:///path/your-custom-config.json
3. Save your logrotate to /path/your-custom-config.json.

Example

In my production setup, I prefer using environment variables as they're easier to override and separate from run scripts. For any Mesos flag, you can always use an environment variable of the form MESOS_SOME_FLAG in lieu of a flag of the form --some-flag.

# Default environment variables for Mesos.
# See http://mesos.apache.org/documentation/latest/configuration/master-and-agent/
cat <<EOF | sudo tee /etc/default/mesos
MESOS_CONTAINER_LOGGER=org_apache_mesos_LogrotateContainerLogger
MESOS_MODULES=file:///etc/mesos-agent-settings.json

EOF

Once logrotation is enabled, write your logrotate configuration for both stdout and stderr streams in the modules JSON file. Note that the logrotate directives need to be separated by a \n, as JSON strings do not support newlines.

/etc/mesos-agent-settings.json

The below configuration sets up to keep the last 4 logs each of size up to 25 MB, compresses rotated logs and doesn't email old logs to any address.

# Default log rotation for all agent machines.
cat <<EOF | sudo tee /etc/mesos-agent-settings.json
{
  "libraries": [{
    "file": "/usr/lib/liblogrotate_container_logger.so",
    "modules": [{
      "name": "org_apache_mesos_LogrotateContainerLogger",
      "parameters": [{
        "key": "logrotate_stdout_options",
        "value": "rotate 4\nsize 25M\nmissingok\nnotifempty\ncompress\ndelaycompress\nnomail\n"
      }, {
        "key": "logrotate_stderr_options",
        "value": "rotate 4\nsize 25M\nmissingok\nnotifempty\ncompress\ndelaycompress\nnomail\n"
      }]
    }]
  }]
}

EOF

Apache Mesos is a "distributed systems kernel" that runs at a higher level of abstraction. It is essentially a cluster manager that provides resource isolation and sharing across distributed applications or frameworks.

My team uses Mesos heavily (yes I know, we aren't on Kubernetes yet because Mesos solves most of my team's problems, is pretty mature and we have a ton of experience running it at scale), and when we decided to upgrade our clusters and rewrite the infrastructure code, one of the features I was excited about using was Fetcher.

Deployment workflow

We use Marathon as our main orchestration platform. It lets us run bundled apps and Docker images on our Mesos cluster and makes it very easy to scale them up or down.

Our usual pattern of deploying apps is —

1. CI builds a Docker image of the app or bundles it up as an archive and pushes it to AWS S3.
2. CD pushes a configuration JSON to Marathon with the --force-deploy flag.

For Docker images, the Marathon configuration is straightforward. For bundled archives, the cmd directive on Marathon is auto-generated by the CD pipeline to look like this —

s3cmd get s3://app-name/app.tgz && tar -xzf *.tgz && ./bin/start.sh

This meant the overall cmd directive was just awkward to look at. Also, s3cmd had to be installed on all of our Mesos agents; s3cmd has its own set of weird issues and it is very difficult to upgrade packages on all of the Mesos agent infrastructure.

Enter Mesos Fetcher

Mesos Fetcher natively supports fetching resources from HTTP, HTTPS, FTP & FTPS URIs. Additionally, it supports caching (huge wins in deploy time and S3 transfer costs if you deploy a lot) and auto-extraction (see supported formats) of resources. If a local Hadoop client is installed, it can also fetch resources from HDFS & S3; this last bit is what we're interested in.

Installing Hadoop

To support fetching S3 URIs, let's first install Hadoop on our Mesos agent and set it up so it is accesible inside the container sandbox. Ideally, this should be baked in your AMI for Mesos agent machines.

The code is heavily documented inline, explaining why we do each of these steps.

#!/bin/bash
# Installation & configuration of Hadoop & Hadoop AWS Tools for Mesos agent.
# Battle-tested on Ubuntu 18.04.

set -e

export HADOOP_VERSION="3.2.0" # See https://hadoop.apache.org/releases.html for latest version
export BASE_DIR="/apps"
export HADOOP_APP_DIR="${BASE_DIR}/hadoop-${HADOOP_VERSION}"
export HADOOP_LOG_DIR="/logs/hadoop"
export MESOS_FETCHER_CACHE_DIR="/data/cache/mesos"

sudo mkdir -p ${HADOOP_APP_DIR} ${HADOOP_LOG_DIR} ${MESOS_FETCHER_CACHE_DIR}

# Download & extract Hadoop binary release.
cd ${HADOOP_APP_DIR}
    # Mirrors are at https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
    sudo wget -O hadoop.tar.gz http://mirrors.estointernet.in/apache/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
    sudo tar -xzf hadoop.tar.gz --strip-components=1 -C .
    sudo rm hadoop.tar.gz
    # Cleanup unneccessary items to keep the Hadoop installation lean.
    sudo rm -rf *.txt
    sudo rm -rf share/doc
cd -

# Set up $JAVA_HOME for Hadoop.
# This locates the JAVA_HOME and updates it in Hadoop's environment file.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
sudo sed -i "s@# export JAVA_HOME=.*@export JAVA_HOME=${JAVA_HOME}@g" ${HADOOP_APP_DIR}/etc/hadoop/hadoop-env.sh

# Turn on Hadoop-AWS optional tools.
# We need this to be able to fetch s3://, s3a:// and s3n:// URIs.
sudo sed -i "s@# export HADOOP_OPTIONAL_TOOLS=.*@export HADOOP_OPTIONAL_TOOLS=\"hadoop-aws\"@g" ${HADOOP_APP_DIR}/etc/hadoop/hadoop-env.sh

# Ask Hadoop to load the AWS SDK.
# This assumes '/root' is $HOME.
# If the default user in your Marathon is not 'root' (as it should be), change this to the home directory of that user.
cat <<EOF | sudo tee /root/.hadooprc
hadoop_add_to_classpath_tools hadoop-aws

EOF

# Add Hadoop to $PATH and customize a few parameters.
# $PATH is available inside each container sandbox, making fetcher work.
# HADOOP_HOME will be picked up by Mesos in place of the --hadoop-home agent flag.
cat <<EOF | sudo tee /etc/profile.d/A00-add-hadoop.sh
export PATH="$PATH:${HADOOP_APP_DIR}/bin"
export HADOOP_HOME="${HADOOP_APP_DIR}"
export HADOOP_LOG_DIR="${HADOOP_LOG_DIR}"
export HADOOP_ROOT_LOGGER=WARN,console

EOF

# Set up Hadoop executables to be discoverable for all users.
# This is so that just running 'hadoop' will work for everyone.
sudo sed -i -e "/ENV_SUPATH/ s[=.*[&:${HADOOP_APP_DIR}/bin[" /etc/login.defs
sudo sed -i -e "/ENV_PATH/ s[=.*[&:${HADOOP_APP_DIR}/bin[" /etc/login.defs

Additional Agent configuration

Once the Hadoop is set up, make sure to set sensible values for these 2 Agent configuration flags —

1. --fetcher-cache-size flag or MESOS_FETCHER_CACHE_SIZE environment variable.
2. --fetcher-cache-dir flag or MESOS_FETCHER_CACHE_DIR environment variable.

For example —

MESOS_FETCHER_CACHE_SIZE=1GB
MESOS_FETCHER_CACHE_DIR=/data/cache/mesos

Marathon configuration

Once the agents are deployed with the new Fetcher & Hadoop setup done, you can change your Marathon deployment configuration from —

{
  "cmd": "s3cmd get s3://app-name/app.tgz && tar -xzf *.tgz && ./bin/start.sh"
}

to —

{
  "fetch": {
    "uri": "s3a://app-name/app.tgz",
    "extract": true,
    "cache": true
  }
}

Apache Thrift allows writing an interoperable type-safe software stack. It comes with a code generation system that has its own definition language that can be converted to code across many programming languages.

As an example, you can write a User data structure that looks like this in Thrift and it can be used to auto-generate code in any language of your choice.

#User.thrift
namespace java com.httgp.models.thrift

struct User {
   1: required i16 id
   2: required string name
   3: optional string nickname
}

The auto-generated TypeScript definition for this Thrift model looks like this —

// User.ts
export interface IUserArgs {
    id: number;
    name: string;
    nickname?: string;
}

export class User {
    public id: number;
    public name: string;
    public nickname?: string;
    constructor(args: IUserArgs) {
        if (args != null && args.id != null) {
            this.id = args.id;
        } else {
            throw new thrift.Thrift.TProtocolException(thrift.Thrift.TProtocolExceptionType.UNKNOWN, "Required field[id] is unset!");
        }
        if (args != null && args.name != null) {
            this.name = args.name;
        } else {
            throw new thrift.Thrift.TProtocolException(thrift.Thrift.TProtocolExceptionType.UNKNOWN, "Required field[name] is unset!");
        }
        if (args != null && args.nickname != null) {
            this.nickname = args.nickname;
        }
    }
    // ...
}

The real benefit of using Thrift becomes obvious when you try to pass this data on to another service that is written in a different language. In this post, I will talk about how to do Thrift serde in Node.js and some common patterns.

Dependencies

1. thrift for serialization & deserialization.

2. node-int64 to handle 64-bit Ints.

3. Optionally, @creditkarma/thrift-typescript for generating TypeScript definitions from your .thrift files. You can simply run —

thrift-typescript --outDir definitions User.thrift

Reading & writing Thrift data in Node.js

The Thrift documentation is quite sparse when it comes to their language-specific implementations, and after a lot of trial and error, here's how I managed to deserialize Thrift data in Node.js.

If you're consuming Thrift-serialized data (say, from a Kafka topic), the data is probably available in Node.js as a Buffer. The deserializeThrift method shows how to deserialize it —

import { TFramedTransport, TBinaryProtocol } from 'thrift';
import { User } from './definitions/User'; // Generated using @creditkarma/thrift-typescript

/**
 * Serializes native data of given model into Thrift.
 * @param data Data to serialize.
 * @param thriftModel Thrift model.
 */
function serializeThrift(data: object, thriftModel: any): any {
    const buffer = Buffer.from(JSON.stringify(data));
    const tTransport = new TFramedTransport(buffer);
    const tProtocol = new TBinaryProtocol(tTransport);
    const serializedData = data.write(binaryProt);
    return serializedData;
}

/**
 * Deserializes Thrift data with given model.
 * @param data Thrift data.
 * @param thriftModel Thrift model.
 */
function deserializeThrift(data: Buffer, thriftModel: any): any {
    const tTransport = new TFramedTransport(data);
    const tProtocol = new TBinaryProtocol(tTransport);
    const deserializedData = thriftModel.read(tProtocol);
    return deserializedData;
}

Deserialization

// rawData = getFromExternalDataSource(...);
const userObject = <User>deserializeThrift(rawData, User);
console.log(userObject);
// { id: 1, name: 'Ganesh', nickname: 'GP' }

Serialization

const userData: User = { id: 1, name: 'Ganesh', nickname: 'GP' };
const userDataAsThrift = serializeThrift(userData, User);
// Now you can write userDataAsThrift to your output sink (like a Kafka topic).

Handling Int64 values in JSON

While other languages have 64-bit Ints, JavaScript's Number supports only IEEE 754 double-precision floats, which are limited to 53 bits. The node-int64 package helps in handling them seamlessly by returning a custom Int64 object. However, if you wish to convert the Thrift-deserialized JSON into anything else, you'll need to manually handle Int64.

Fortunately, JSON.stringify() takes a "replacer" parameter that you can use to modify its default behaviour.

/**
 * Custom JSON stringify replacer.
 *
 * Converts `Int64` to `Number`. Returns same value if it isn't `Int64`.
 * NOTE: Won't be precise for VERY large numbers.
 */
function customStringifier(key: string, value: any): Number | any {
    if (value instanceof Int64) {
        return value.toNumber();
    } else {
        return value;
    }
}

// Convert deserialized object to a String —
JSON.stringify(deserializedObject, customStringifier);

A lot of programming languages and runtimes (like Go, Rust & C/C++) offer self-contained distributable binaries, and as someone that spends most of their time on Node.js, I've been quite jealous of them... until now.

js2bin aims to build single binaries for Node.js projects on Linux and macOS. Although similar projects like nexe and pkg exist, js2bin is slightly different (IMHO, better) in how it works; in the words of the author —

They (nexe and pkg) both embed users application and other content by appending to the executable – while this method works, it can also trip malware scanners as the executable contains extraneous content.

How to use js2bin

To be able to use js2bin, you need to first bundle your Node.js application into a single JS file. You can try using webpack or rollup, but I found a much simpler solution in ncc.

ncc supports TypeScript out of the box too, so all you have to do to is run —

ncc build src/index.ts -o bundle

src/index.ts is your main entrypoint, and bundle is the output directory.

Once you have a single JS file, you can then run —

js2bin --cache --build --platform=darwin --platform=linux --platform=windows --app=bundle/index.js

This creates binaries for macOS, Linux and Windows using the bundled JS you created in the previous step.

🚩 Caveats

js2bin does not work with projects that have native modules. To quickly check if your project uses native modules, run —

find node_modules -type f -name "*.node"

Real-world example

I used js2bin to generate executables and a CD workflow to automatically attach them to releases on a hobby project of mine; here's the key pieces of interest —

1. npm scripts
2. GitHub Actions CD workflow
3. Releases page with binaries as release attachments.

I've ranted about this before, but I wanted to take another long and hard look at this whole situation — phone manufacturers removing the headphone jack.

I shall now proceed to talk about how fucking stupid this is!

ಠ_ಠ Apple

Granted, Apple isn't the first major OEM to remove the headphone jack — that dubious distinction actually belongs to Oppo. Back in 2012, Oppo launched the Finder without a headphone jack and bundled micro-USB earphones with the phone; they discontinued it after a year on the market, probably realizing what a colossal mistake that was. However, Chinese OEMs did not give up on the idea and continued experimenting with removing it; most notably Oppo (again, smh), Vivo and LeEco. They were all just that - experiments, until Apple decided to ditch the jack in the iPhone 7. And they had the balls to call it "courage".

The problem with Apple making any big decision like this is, they have a humongous clout of influence and have little real competition. If an Android OEM removed the headphone jack, consumers can pick another. Consumers that want the Apple experience have no choice but to accept whatever Apple gives them.

Android see, Android do

At the original Pixel's launch event, Google took a not-so-subtle jab at Apple for ditching the headphone jack. This was actually on their marketing material —

3.5mm headphone jack satisfyingly not new

Only a year later, Google pulled an Apple and released the Pixel 2 without a headphone jack. Why? For the glory of Satan, of course. Or probably because everyone follows in Apple's footsteps, for good or bad.

Why they're doing this to us

With increasing unrest amongst the masses, various OEMs have explained why they removed the headphone jack.

1. Google product chief Mario Queiroz told TechCrunch —

“The primary reason [for dropping the jack] is establishing a mechanical design path for the future”.

2. Apple Senior VP of hardware Engineering Dan Riccio told BuzzFeed —

“It was holding us back from a number of things we wanted to put into the iPhone... It was fighting for space with camera technologies and processors and battery life. And frankly, when there’s a better, modern solution available, it’s crazy to keep it around”.

3. During the same interview, Apple VP of iOS, iPhone & iPad marketing Greg Joswiak said —

“The audio connector is more than 100 years old... It had its last big innovation about 50 years ago. You know what that was? They made it smaller. It hasn’t been touched since then. It’s a dinosaur. It’s time to move on”.

Why this is anti-consumer

1. On wanting to make thinner phones.

Unless they are making a phone with titanium or polycarbonate, OEMs must stop making thinner phones. And don't even get me started on devices with glass backs. How often do you run into a person that says "Man, I wish my phone was thinner"? If you are one of those mythical creatures, fight me 1v1. Also, look at the Galaxy S8 - it packed in more technology AND a headphone jack at almost the same thickess as the iPhone 7.

2. On calling the technology "old".

It has lasted a 100 years probably because it works. And they're ubiquitous for that reason - they're on computers, laptops, 99% of current mobile phones, car audio systems, ATMs, credit card readers and a whole range of accessories!

3. On Apple calling it "courage".

Apple has historically been on the forefront of ditching obsolete technology - removing floppy drives, CD/DVD drives, etc. It made sense to remove those because they had better alternatives. The headphone jack, on the other hand, does not have a better alternative! So no, Apple, this isn't courage; your opinion is bad and you should feel bad.

4. On wireless being the future.

Bluetooth headphones come with a whole bunch of problems - worse audio quality due to compression, stuttering/skipping, pairing (not to mention proprietary solutions like Google's Fast Pair only worsening vendor lock-in) and limited battery life. Remember that time when headphones didn't need charing or even a battery?

5. On living with the new USB-C to AUX dongle.

I'd have to carry a dongle in my pocket now? The dongle has a DAC too, because fuck analog, right? It is just one additional point of failure. What happens when you lose the dongle? What happens when you want to charge your phone and listen to music at the same time?

Light at the end of the tunnel

A lot of people have talked about the problems with this transition, but there might be some positives to come out of this.

1. Bluetooth 5.0 and aptX HD hopefully improve the shitty audio quality that Bluetooth offers today.
2. Better waterproof devices - the iPhone 7+ and Pixel 2 are both IP67 rated, admittedly because they got rid of the headphone jack. But then the Galaxy S8 also has a IP67 rating and a headphone jack, so ¯\_(ツ)_/¯.
3. More room for a bigger battery!

While I'm being cautiously optimistic of this change, I will hold on dearly to my headphone jack-equipped phone and all of my headphones. Apple and Google can pry them away from my cold dead hands.