Encrypting internal RESTful requests between micro services

[chenlica]

There are many internal RESTful requests between micro services in the Texera system. We want to know whether we need to encrypt these requests to make them more secure.

@parshimers @madhusudancs and others: any thoughts?

[parshimers]

I think between services, there should be encryption, one way or another. It can easily be the case that different services are on different machines. In this case it can be important not only for privacy, but for stopping someone from injecting spoofed messages between the two services.
How this is done more concretely, I am a little less certain of. Asterix implements the ability to have certificate auth between all nodes in the cluster, which wraps all the IPC. However this was implemented before k8s was popular and basically assumes it can be running on bare metal. If you assume everything will always be running in k8s if it's distributed, then I think you can use a service mesh like istio that presents a proxy on each pod that manages the encryption separately. Which would be a lot more convenient than having to add this functionality internally and figure out some way to deal with all the certificate management. I'd be interested to hear Madhu's take on that idea however.

[dzueck]

I have been able to show that using packet sniffing tools like tcpdump and Wireshark, the traffic between services does leak potentially private information such as data set names and contents. There is likely more, but I did not look for any more. In my case I had root access to the deployment to view the tcp communications, however as parshimers mentioned, you would not need root privileges to read these packets if services are spread out on multiple machines. In this setup you would only need a separate device on the same network as one of the services.

[aglinxinyuan]

Are you assuming that it's http? What if we use https? I don't know much about how does tcp layer work, but I guess the certificate can provide encryption at application layer.

[dzueck]

I am not assuming we are using http. I have shown that we are and that using http can leak information. https does seem like a valid solution, however we need to figure out how to do certificate distribution as https requires an authority to authenticate connections. Additionally, https would provide even more important qualities like integrity and authentication which we have not discussed much here, but are also very important.

[dzueck]

Further looking into this, I believe that https is actually not what we are looking for here. The important features of https we need are:

1. Encryption to prevent external parties from reading sensitive information
2. Authentication to prevent man in the middle attacks where a third party pretends to be a CU or micro service. This would allow them to request sensitive data or provide fake data among other things.
3. Integrity to prevent external parties from modifying requests to provide fake data/requests.

While https provides all 3 of these, they are actually provided by TLS which https uses. For this reason we should be discussing providing TLS between containers instead. There are a few existing tools to do this (often called service meshes), however the most light weight and simple one I could find is linkerd which I will be looking into using.

Note: Due to the proposal to utilize VMs #8430 for container runtimes, any solution we choose to use should utilize sidecars instead of the sidecar less modern approaches. This is due to the fact that vms use a separate kernel with a separate networking stack which can interfere with non sidecar approaches.

[parshimers]

I agree with Dylan's points about focusing on wrapping everything (TLS/mTLS) rather than focusing solely on HTTP. I think giving linkerd a detailed look is also great direction. The earlier we can get something to work the better; it is going from a 0 to a 1.

[chenlica]

@dzueck If you can include some diagram, that will be helpful.