Setup Hashicorp Vault in EC2 and Connect to Mulesoft

 This blog will take you through to create your own Hashicorp Vault Server in EC2 so that you can push property from this vault to your Mulesoft App

 Hashicorp Vault, Vault Project, Mulesoft, Mule Properties, Vault Mule Properties

  Mulesoft    |      Anupam Chakraborty    |      Jun 27 2020 05:08 PM

 Login to Like      Login to Follow      235 Views

Introduction

Today, I will explain how to set up a simple Hashicorp Vault in an EC2 Server and then access it from our computer so that it can provide properties to a Mule Application. There is amazing documentation around this in the Vault Project Website, however, this would be more of a step by step code with exactly what you need to get going. 

Prerequisite

  • Amazon AWS Account to build an EC2 system. I will use a Basic T2.micro Free tier EC2 System. Take a close look at your AWS account to make sure you are not incurring charges. 
  • Mule ESB running in Local Computer.

So, without any further ado, let us build our Vault and Connect to Mule.

Step 1: Setting up a t2.micro EC2 Server.

I will not go through the steps for how to bring up an EC2 Server. If you need help around this. Look at my previous blog Amazon EC2 Cloud with LAMP Stack and Loopback CRUD API. I will use the Amazon Linux 2 AMI (HVM), SSD Volume Type with t2.micro Instance, and use the standard 8 GiB Storage. The important part is the security group. We will use the SSH on port 22 along with a custom TCP port 8200 for this box.


A screenshot of a cell phone

Description automatically generated

Once this is up and running, we will SSH into this instance using an SSH key.


For me this came as

$ ssh -i "SlalomEC2.pem" ec2-user@ec2-3-87-191-61.compute-1.amazonaws.com

Because we are login for the first time, we would update the system by running

$ sudo yum update

Step 2: Install Vault in our EC2


Next, we will download 2 packages: the vault itself and consul for the backend from the respective hashicorp website. They are downloaded as zip files. We will then extract them.

$ wget https://releases.hashicorp.com/vault/1.4.2/vault_1.4.2_linux_amd64.zip
$ wget https://releases.hashicorp.com/consul/1.8.0/consul_1.8.0_linux_amd64.zip

$ unzip vault_1.4.2_linux_amd64.zip
$ unzip consul_1.8.0_linux_amd64.zip

at this time, there would be 4 files in my home directory

A close up of a screen

Description automatically generated

Then we would move them to the bin folder so that they can be executable. Finally, we will delete the zipped file that was downloaded.


$ sudo mv vault /usr/local/bin/vault
$ sudo mv consul /usr/local/bin/consul

$ rm vault_1.4.2_linux_amd64.zip
$ rm consul_1.8.0_linux_amd64.zip

At this time Vault is running, we can verify this with the command


$ vault version
Vault v1.4.2

Step 3: Setup the Vault Config


Now let us set up our Vault so that we can connect it from our computer. 

We need to create a file name something like vaultConfig.hcl


$ touch vaultConfig.hcl
$ nano vaultConfig.hcl


And fill up the file with the following text


listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}

ui = true

storage "consul" {
address = "127.0.0.1:8500"
path  = "vault/"
}

disable_mlock=true

you can make sure if you have set up the file correctly by issuing the following command

$ more vaultConfig.hcl


A screenshot of a cell phone

Description automatically generated


Step 4: Bring up the Vault Backend

If you see the above config, we would be using the consul as our storage. We can start vault without any backend, and it will store the data in the memory. However, we would rather use the Consul storage, which is much more reliable and is production-ready.


To start the consul, simply issue the command

$ consul agent -dev &


The & at the end will make sure that we are running the consul in backend and hence, even if we break the script using Ctrl+C, the consul will keep running. This can be checked using the following command

$ ps -ef | grep consul


Step 5: Bringing up the Vault Server.


With the configuration in place, starting the server is simple, as shown below. Modify the -config flag to point to the proper path where you saved the configuration above.


$ vault server -config=vaultConfig.hcl &
==> Vault server configuration:

Api Address: http://127.0.0.1:8200
Cgo: disabled
Cluster Address: https://127.0.0.1:8201
Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Recovery Mode: false
Storage: consul (HA available)
Version: Vault v1.4.2

==> Vault server started! Log data will stream in below:


Yohoo, we have the vault up and running now. But wait, it is a vault. So, we would need to unseal it.


Step 6: Unseal the vault


The next step is to unseal the vault. To do that, we should first set up the Vault Server as an Environment variable and then need to use the following command.


$ export VAULT_ADDR='http://127.0.0.1:8200'

$ vault operator init

Unseal Key 1: qz28vdxHN/3RvJ7KlDInFDpZQCKkr0RgeYIDTQJu3glY
Unseal Key 2: msJbFKKtf7tN0Vb7hs+yEBXE20wz1Syqlirp3iUcGYDA
Unseal Key 3: TAeEbw5XjEhzzBckmkz/DNjWcJ7u551gSowyEfTFdKyO
Unseal Key 4: CUt1uKw+mmbBCImgPC1o8r0N3Z7VhgwKDw7Vo19lu9O5
Unseal Key 5: XRJodDat7J9qkvgQ3fF4IYcfpd1b3gHSMJQgRmVA/Osd

Initial Root Token: s.1PUGwN7lQ9ZGWGTJ5nZKunms

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.


We should keep a note of these keys, this would be used to unseal our vault.

For that, we should execute the command vault operator unseal 3 times with 3 of the key above.


$ vault operator unseal
Unseal Key (will be hidden):
Key-Value
---  -----
Seal Type  shamir
Initialized  true
Sealed true
Total Shares  5
Threshold  3
Unseal Progress  1/3
Unseal Nonce  7df52188-ed15-1337-7b39-b73bb3d03489
Version  1.4.2
HA Enabled  true


$ vault operator unseal
Unseal Key (will be hidden):
Key  Value
---  -----
Seal Type  shamir
Initialized  true
Sealed  true
Total Shares  5
Threshold  3
Unseal Progress  2/3
Unseal Nonce  7df52188-ed15-1337-7b39-b73bb3d03489
Version  1.4.2
HA Enabled  true
$ vault operator unseal
Unseal Key (will be hidden):

[INFO] core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201
[INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
[INFO]  core: vault is unsealed
[INFO]  core: entering standby mode

Key  Value
---  -----
Seal Type  shamir
Initialized  true
Sealed  false
Total Shares  5
Threshold  3
Version  1.4.2
Cluster Name  vault-cluster-95c51d21
Cluster ID  25d8d6a8-86d6-8320-3e2d-1545e291d59b
HA Enabled  true
HA Cluster  n/a
HA Mode  standby
Active Node Address  <none>
[ec2-user@ip-172-30-0-151 ~]$ 2020-06-27T03:00:52.554Z [INFO]  core: acquired lock, enabling active operation


Our vault is now unlocked and ready to be used.

Step 6. Login to vault and create a Secret Engine


We should remember the token that we received when we used the command vault operator init. This token will be used to login to the vault. So in my case, I would use the following command


$ export VAULT_ADDR='http://127.0.0.1:8200'
$ vault login s.1PUGwN7lQ9ZGWGTJ5nZKunms

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key  Value
---  -----
token    s.1PUGwN7lQ9ZGWGTJ5nZKunms
token_accessor  n3IvnysY1REGdZAPaHndI1Cj
token_duration  ∞
token_renewable  false
token_policies  ["root"]
identity_policies  []
policies  ["root"]


Create a KV-V2 Secret Engine


Now, that I have logged in, I should create a Secret Engine so that I can use the secret engine in my Mule Project. 

We have to create a secret engine in the vault to store different secrets. If we use any other secret engine, we would get an error with code 403.

So, let us use the following command to create a secret engine.


$vault secrets enable -path=mySecret -description='This is the demo Secret Engine' kv-v2
Success! Enabled the kv-v2 secrets engine at mySecret/


This should create a key value version 2 secret engine, that we would use for our requirement.

You can check all the secret engines in the vault system using the following command.


[ec2-user@ip-172-30-0-151 ~]$ vault secrets list
Path  Type  Accessor  Description
----  ----  --------  -----------
cubbyhole/  cubbyhole  cubbyhole_cee084dc  per-token private secret storage
identity/  identity  identity_40dc37fc  identity store
mySecret/  kv  kv_ee4a9dce  This is the demo Secret Engine
sys/  system  system_292962f2  system endpoints used for control, policy and debugging


Step 7: Put and Get Secret from the Engine


Now that we have created a secret engine, let us first create a test secret. I will call it muleMessage.


$vault kv put mySecret/muleMessage welcomeMsg='Welcome to MuleSoft'
Key  Value
---  -----
created_time  2020-06-27T03:15:44.972268511Z
deletion_time  n/a
destroyed  false
version  1


You can see this using the following command

$ vault kv get mySecret/muleMessage
====== Metadata ======
Key  Value
---  -----
created_time  2020-06-27T03:15:44.972268511Z
deletion_time  n/a
destroyed  false
version  1

======= Data =======
Key  Value
---  -----
welcomeMsg  Welcome to MuleSoft


You can also, browse the secret like this:


$vault kv list mySecret
Keys
----
muleMessage


You can get a specific value directly using this command


$ vault kv get -field=welcomeMsg mySecret/muleMessage
Welcome to MuleSoft


Now let us create another secret with 2 value.


$vault kv put mySecret/httpListener host=0.0.0.0 port=8081
Key  Value
---  -----
created_time  2020-06-27T03:20:47.156553753Z
deletion_time  n/a
destroyed  false
version  1


If you do a get on this now

$ vault kv get mySecret/httpListener
====== Metadata ======
Key  Value
---  -----
created_time  2020-06-27T03:20:47.156553753Z
deletion_time  n/a
destroyed  false
version  1

==== Data ====
Key  Value
---  -----
host  0.0.0.0
port  8081



Step 8: Accessing the vault from our computer.


To access the vault from our computer, we should install the vault from the hashicorp website: https://www.vaultproject.io/downloads. This has a download for different operating systems. Once we have installed this on our computer. If we execute the following command, we should be able to access the secrets directly from our computer instead of SSH.


$ export VAULT_ADDR='http://<EC2 IP Address>:8200'


In my case, I had to do like this


anupam.chakraborty@mbpr ~ % export VAULT_ADDR='http://3.87.191.61:8200'
anupam.chakraborty@mbpr ~ % vault login s.1PUGwN7lQ9ZGWGTJ5nZKunms
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key  Value
---  -----
token  s.1PUGwN7lQ9ZGWGTJ5nZKunms
token_accessor  n3IvnysY1REGdZAPaHndI1Cj
token_duration  ∞
token_renewable  false
token_policies  ["root"]
identity_policies  []
policies  ["root"]


Now you should be able to hit all the kv get and put from your pc directly.


anupam.chakraborty@ mbpr ~ % vault kv get mySecret/httpListener
====== Metadata ======
Key  Value
---  -----
created_time  2020-06-27T03:20:47.156553753Z
deletion_time  n/a
destroyed  false
version  1

==== Data ====
Key  Value
---  -----
host  0.0.0.0
port  8081


Step 8: Accessing the UI

Here comes the best part of this vault. There is a Web-based user interface available for us to use. Simply go to http://<EC2 IP Address>:8200/ui


A screenshot of a cell phone

Description automatically generated


This gives a nice frontend and you should be able to access all the vaults that you have created using this UI.


A screenshot of a cell phone

Description automatically generated


Step 9: Connect to the Vault from MuleSoft

Now, we would use the Vault to pass the properties to our mule app.

In order, to do this we would have to use the Vault Property Provider connector. The source code is available here at https://github.com/avioconsulting/mule-vault-properties-provider.


We can clone this and then publish this into our exchange. Then finally add the dependency using the following pom xml


<dependency>    
<groupId>${orgId}</groupId>   
<artifactId>vault-connector</artifactId>   
<version>1.0.0-SNAPSHOT</version>   
<classifier>mule-plugin</classifier>
</dependency>


And here is the simple flow to use this vault

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:vault-properties-provider="http://www.mulesoft.org/schema/mule/vault-properties-provider" xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
           xmlns:http="http://www.mulesoft.org/schema/mule/http"
           xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd
http://www.mulesoft.org/schema/mule/vault-properties-provider http://www.mulesoft.org/schema/mule/vault-properties-provider/current/mule-vault-properties-provider.xsd">

           <vault-properties-provider:config name="Vault_Config" doc:name="Vault Config" >
                       <vault-properties-provider:token-connection vaultUrl="http://3.87.191.61:8200" engineVersion="v2" vaultToken="s.1PUGwN7lQ9ZGWGTJ5nZKunms" />
           </vault-properties-provider:config>

           <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" basePath="">
                       <http:listener-connection host="${vault::mySecret/httpListener/host}" port="${vault::mySecret/httpListener/port}" />
           </http:listener-config>

          
           <flow name="homeFlow" >
                       <http:listener doc:name="Listener" doc:id="6a565a15-8dc4-44d4-92b3-4f39c1c7d7e2" config-ref="HTTP_Listener_config" path="/sayHi/{name}"/>
                       <set-variable value="${vault::mySecret/muleMessage/welcomeMsg}" doc:name="prop" variableName="prop"/>
                       <ee:transform doc:name="Transform Message" >
                                   <ee:message >
                                               <ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
           msg: "Hi " ++ ( attributes.uriParams.name default "Jon Doe") ++ ". " ++
                       (if (now().hour >= 12 ) "Good Afternoon" else "Good Morning") ++
                       ". "  ++ vars.prop
}]]></ee:set-payload>
                                   </ee:message>
                       </ee:transform>
           </flow>
</mule>


Finally Thank you for going through this blog. Let me know if you were able to make this to work. Thank you once again for reading my blogs.



Comments:

Leave a Comment:

Please login to post comment into this page.

Please login to submit a new Blog.