How did I develop a crypto service for Tableau Server

For more information about Tableau Server, you can read my blogs here

What is this project about?

Tableau Server is an on-premise data analytics and visualization platform. Customer can deploy bits in their own data center, or in AWS, Azure, and GCP.

There are two categories of passwords within a Tableau Server cluster:

  • User input passwords, which are entered by users
  • AWS key
  • smtp passwords
  • Windows runas user password
  • Server managed passwords, which are randomly generated by server

  • jdbc password
  • Postgres admin password
  • ZooKeeper password

So the passwords I’m talking about is not the password for someone’s server account. The latter ones live in a table of the cluster’s PostgreSQL database and are already encrypted.

The current version of Tableau Server (v10) stores all the passwords in plain text in a configuration file, which is distributed among machines.

The new version of Tableau Server also stores all passwords in plain text, but in a Znode in ZooKeeper, as I described in this blog.

The project is about developing a crypto service that encrypts passwords before persisting them somewhere, and decrypts passwords into memory when server uses them, so that no more clear text secrets live in Tableau Server. The new service should support both our old monolith JRuby app and our new service-oriented server architecture.

Why are we developing this feature?

Tableau Server is for enterprise customers, and the top 3 things that enterprise IT cares about are SECURITY, SECURITY, SECURITY.

According to our sales team, keeping secrets in clear text has been one of the biggest sales blockers for Tableau Server. Our customers have also been bringing up many times that either their huge concerns w.r.t the potential password leaks, or the friction they’ve experienced to buy Tableau Server – they need to get an exception from IT since Tableau Server is the only app in house that don’t encrypt passwords.

In a nutshell, Tableau Server is not enterprise-ready as long as it doesn’t support password encryption.

Trade-offs

There’s no doubt that we are gonna build the crypto service for new Tableau Server, a service-oriented server architecture that will come out in 2017. The biggest trade-off we faced was whether to put efforts on enabling this feature in our old monolith JRuby server app.

The downsides of doing that are:

  • Lots of efforts required. Our old server architecture is a monolith JRuby app with messy logic (yeah, everyone agrees on it! It’s totally messed up because people have been keeping on adding new stuff here and there for 9 years – the ruby app started on 2007). The crypto service will be written in Java and incorporate it into old server can be really hard.
  • Uncertain schedule. Because of the large amount of work expected, no one is certain whether we can finish this feature, fully test it, and ship it on time. If it comes out too late, say only three months before new server gets shipped, it’s probably much less valuable.
  • We are not sure if it worths it. If our customers have been living with plain text passwords for 9 years, they probably can wait for the new server for another six months?
  • Developer frastruction. Not only because it’s a messy job, but also because you know your code will be deleted a year later.

The upsides of doing that are:

  • Removes the sales blocker as soon as possible
  • Relieves customers pain and complaints as soon as possible
  • Or at least demonstrate to our customers that we have heard of their requirements and we are executing on it (this is sounds very much self-comforting, but it’s been brought up serveral time. Yeah, it’s all about ATTITUDE)

The team eventually decided to do it.

Design

Compared to the complicated integration of crypto service and our ruby app, the design of crypto service itself is actually much more straightforward.

The core idea is:

Encrpt actual secrets with an encryption key, and encrypt the encryption key with a master key

Architecture

File System / Config Dir / Config File
------yml file
| inputs
| - master key algorithm
| - encryption key algorithm
| - keystore location
|
| generated values
| - master key id
| - encryption key
| - encryption key id
------

File System / Config Dir / Java KeyStore (JKS)
-----------
| Master |
| key       |
-----------

Bootstrap

  1. Provide the following three args:

– master key algorithm
– encryption key algorithm
– keystore location

  1. Use master key algorithm to create a master key, and put it in Java Keystore (JKS) in keystore location. JKS will be in Tableau Server’s config directory, and will be guarded by the dir’s access permission.

  2. Create encryption key using encryption key algorithm, encrypt it with master key, and put the encrypted value in config file.

  3. Encrypt all existing passwords in config file

When encrypting passwords

Whenever a new key-value pair is requested to be put into config file:

  1. Check if the key is in our encryption list. If true, proceed; otherwise, break out and just put them as clear text in config file
  2. Get encrypted encryption key value, decrypt it with master key to get the actual encryption key. You need to have access to the JKS file to read master key.
  3. Encrypt passwords with encryption key

When decrypting passwords

  1. Get encrypted encryption key value, decrypt it with master key to get the actual encryption key. You need to have access to the JKS file to read master key.
  2. Decrypt passwords with encryption key

Key-rolling

I made key rolling an atmoic transaction, so that it can recover from any failure.

  1. Take a backup of the current config file and the JKS, put them in a temp dir which shares the same access permission as the config dir
  2. Roll all server-managed passwords, encrypt them with existing crypto service, and persist new values in config file
  3. Decrypt all passwords (both server-managed and user-managed) in config file, and hold them as clear text in memory
  4. Force creating a new JKS with new master key
  5. Create a new encyption key
  6. Reinitialize crypto service with new crypto properties
  7. Use the new crypto service to encrypt all password held in memory
  8. Put all new crypto properties and all encrypted passwords into the config file
  9. If any error happens, force restoring the config file and the JKS in your backup

When to roll keys

Here are some best practices I recommend:

  • When a worker machine is removed/decommissioned from your cluster
  • Every a few months
  • Whenever the admin think is necessary

My Role in this project

I played two major roles in this project

1. Fireman

I was not an orginal team member of this project. This project was started in parallel with my ZooKeeper project. When I finished mine, my manager asked me to join my colleague in this project.

At that point, probably some time in June, everyone was positive about the project progress and thought it was on schedule. Adding me is just to help the project move faster. After I joined, and gave an estimation of all backlogs, I realized that we were actually far behind the schedule, even two of us were not able to finish it on time.

I brought this up to my manager, and had all our devs, testers, and PM sit down and evaluate our backlogs together. It turned out we WERE at an extreme risky schedule. I didn’t know why my colleague (the only original developer for this project) didn’t warn everybody.

2. One of the Key Contributor

I am one of the main contributors, and finish the following works:

  • crypto service bootstapping in both old Jruby app and new service-oriented Java app
  • key rolling
  • most Java-Ruby integration
  • enabling unit and integration tests
  • all customer-facing commands

Most Important Stuff I learned from this project

Project Management:

  • Always keep an eye on the project schedule, and don’t be shy to alert everybody when you cannot finish your work on time
    Tech:

  • Learned quite a lot crypto architecture and related knowledge

  • Learned a lot Ruby and JRuby
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s