VGS Developer's Guide

An overview of VGS developer tools and an example integration guide.

The VGS Dashboard’s route editing UI provides a good starting point for setting up your routes, but as the config grows in complexity, we recommend integrating the VGS route into your project’s codebase. Luckily, all VGS routes are available in YAML format and can be stored in your project's version control system.

In this guide we will go through the process integrating VGS routes files into an existing dockerized application, and using existing VGS developer tools to test/deploy changes to your VGS vault in parallel with changes to the main application.

---

IMPORTANT

This guide assumes that you have already created your VGS account, and have completed the initial setup guides with Payment Processing - Pay Facilitation specified as a primary use case.

---

Structure

In this guide, we’re going to cover:

1. How to install and set up vgs-cli and vgs-satellite;
2. How to download your VGS vault’s routes and test them locally;
3. How to integrate your VGS routes' changes into your existing CI/CD pipeline;

Full code for these examples can be found on our github.

Setup

Both vgs-cli and vgs-satellite are available as docker images, so I’m going to focus on the method of running these tools through docker-compose. If you’d like to install and run these apps manually, please follow the links to their respective pages.

VGS CLI

VGS CLI is a developer tool that helps you build, test, and manage your configurations in VGS from a terminal. It enables programmatic usage of VGS APIs in code or as a part of CI/CD pipelines.

Let’s start by creating a docker-compose.yaml file and adding our vgs-cli service:

version: '3.2'
services:
  vgs-cli:
    image: quay.io/verygoodsecurity/vgs-cli:${VERSION:-latest}
    command: bash
    stdin_open: true # docker run -i
    tty: true # docker run -t
    volumes:
      - ./:/data/
    ports:
      - '7745:7745'
      - '8390:8390'
      - '9056:9056'

The first half of this configuration is pretty straightforward - we are specifying the image to pull from VGS’s quay repository, which command to run upon start, and enabling interactive shell with stdin_open and tty flags.

In the volumes section we are mapping the current folder to container’s /data folder, enabling us to work with VGS vault’s routes file locally. ports section maps container’s ports to your machine’s ports, for CLI to work correctly.

Now, we can start the CLI container, and authenticate our CLI:

$ docker-compose up -d vgs-cli

Creating network "git-cli-satellite-example_default" with the default driver
Creating git-cli-satellite-example_vgs-cli_1 ... done

$ docker-compose exec vgs-cli vgs login

Can not open your browser, please open this URL in your browser: https://auth.verygoodsecurity.com/auth...

After the second command, you’ll be prompted to follow the link to verify your VGS account. You may also be asked for permission to access your OS key management system (Mac OS Keychain, Linux Secret Service, Windows Credential Vault). VGS CLI uses it for the secure storage of keys used for encryption of your access tokens.

To download your VGS vault’s routes, simply run the following command with your vault id:

docker-compose exec vgs-cli vgs get routes --vault tntfpb2cmsw > ./vgs_routes.yaml

As a result, you should see that your routes file has been downloaded to your local folder:

.
 |-docker-compose.yaml
 |-vgs_routes.yaml

VGS Satellite

VGS Satellite is a developer tool to help you build, test, and verify your integration with VGS directly from your local machine. It comes in 2 flavours: a desktop app, and a docker image. For the purposes of this guide, we’re going to use the latter.

Add the following snippet to your docker-compose.yaml file:

vgs-satellite:
  image: verygood/satellite:${VERSION:-latest}
  ports:
    - 8089:8089
    - 9098:9098
    - 9099:9099
  volumes:
    - $HOME/.vgs-satellite/:/data
    - ./:/routes
  environment:
    # path to routes file that will be loaded upon container start
    SATELLITE_ROUTES_PATH: /routes/vgs_routes.yaml

Again, we’re mapping your current directory to image’s /routes folder, and creating a folder in your $HOME directory to store satellite’s data. SATELLITE_ROUTES_PATH variable contains the location of the routes file within the container, so that it is applied to satellite’s local vault.

We can now start the satellite container by running:

$ docker-compose up -d vgs-satellite

Starting git-cli-satellite-example_vgs-satellite_1 ... done

And test that it’s working by sending a request to it:

$ curl http://localhost:9098/post \
  -H "Content-type: application/json" \
  -d '{"card_number": "4111111111111111", "card_cvc": "1111", "card_exp": "01/01/2025"}'

{
  "args": {},
  "data": "{\"card_number\": \"tok_sat_PzsyygDud8LQWY3pmvvZUb\", \"card_cvc\": \"tok_sat_Ng3mU2pmVn58oJmgq23r9A\", \"card_exp\": \"01/01/2025\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "121",
    "Content-Type": "application/json",
    "Host": "echo.apps.verygood.systems",
    "User-Agent": "curl/7.64.1"
  },
  "json": {
    "card_cvc": "tok_sat_Ng3mU2pmVn58oJmgq23r9A",
    "card_exp": "01/01/2025",
    "card_number": "tok_sat_PzsyygDud8LQWY3pmvvZUb"
  },
  "origin": "185.237.74.238, 10.22.116.69",
  "url": "https://echo.apps.verygood.systems/post"
}

Working with routes

Now, let’s say, that we would like to change the format of our credit card number alias format from VGS Alias to Format Preserving, Luhn Valid (6T4), so that BIN and the last four digits are preserved (e.g. 4111111111111111 becomes something like 4111119381251111). In order to accomplish this, we would need to:

1. Apply changes to our local routes file;
2. Test updated routes locally;
3. Apply updated routes to our vault;
4. Check the vault’s logs to see if everything is working correctly;

Updating the routes

Let’s start by updating the routes. We need to find the portions where card_number is being redacted and revealed in inbound and outbound routes, and change the public_token_generator in both those locations from UUID to FPE_SIX_T_FOUR. You can check our all of the available VGS aliasing formats on our docs.

data:
- attributes:
    created_at: '2021-02-11T13:22:05'
    destination_override_endpoint: https://echo.apps.verygood.systems
    entries:
      ...
      operation: REDACT # <- remove sensitive data
      operations: null
      phase: REQUEST
      public_token_generator: UUID # <- aliasing format, change this to 'FPE_SIX_T_FOUR'
      targets:
      - body
      token_manager: PERSISTENT
      transformer: JSON_PATH
      transformer_config:
      - $.card_number # <- location of sensitive data within payload
      ...
- attributes:
    created_at: '2021-02-11T13:25:02'
    destination_override_endpoint: '*'
    entries:
      ...
      operation: ENRICH # <- restore sensitive data
      operations: null
      phase: REQUEST
      public_token_generator: UUID # <- aliasing format, change this to 'FPE_SIX_T_FOUR'
      targets:
      - body
      token_manager: PERSISTENT
      transformer: JSON_PATH
      transformer_config:
      - $.card_number # <- location of sensitive data within payload
      ...

Testing the routes

Before we can test these changes, we have to restart vgs-satellite container:

$ docker-compose kill vgs-satellite

Killing git-cli-satellite-example_vgs-satellite_1 ... done

$ docker-compose up -d vgs-satellite

Starting git-cli-satellite-example_vgs-satellite_1 ... done

Now, send a test request:

$ curl http://localhost:9098/post \
  -H "Content-type: application/json" \
  -d '{"card_number": "4111111111111111", "card_cvc": "1111", "card_exp": "01/01/2025"}'

{
  "args": {},
  "data": "{\"card_number\": \"4111115400611111\", \"card_cvc\": \"tok_sat_Ng3mU2pmVn58oJmgq23r9A\", \"card_exp\": \"01/01/2025\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "107",
    "Content-Type": "application/json",
    "Host": "echo.apps.verygood.systems",
    "User-Agent": "curl/7.64.1"
  },
  "json": {
    "card_cvc": "tok_sat_Ng3mU2pmVn58oJmgq23r9A",
    "card_exp": "01/01/2025",
    "card_number": "4111115400611111"
  },
  "origin": "185.237.74.238, 10.22.113.146",
  "url": "https://echo.apps.verygood.systems/post"
}

As you can see, card_number still passes as a valid credit card number, although different from the one we sent.

Okay, now, let’s automate the testing process by creating some simple Python tests route_tests.py that will send requests to your local vgs-satellite, and verify that the payload has been redacted/revealed:

import os
import re
import urllib3

import requests

class TestVgsRoutes:
    reverse_proxy_port = '9098'
    forward_proxy_port = '9099'
    satellite_url = os.environ.get('VGS_SATELLITE_URL', 'http://localhost')
    headers = {
        'Content-type': 'application/json',
    }
    data = {
        'card_number': '4111111111111111',
        'card_cvc': '1111',
        'card_exp': '01/01/2025'
    }

    def test_both_routes(self) -> None:
        """
        Test that the inbound data is being redacted and conforms to the specified alias format.
        Then, test that the outbound data is being revealed.
        """
        # https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
        urllib3.disable_warnings()

        inbound_response = requests.post(f'{self.satellite_url}:{self.reverse_proxy_port}/post',
                                         headers=self.headers,
                                         json=self.data)
        inbound_json = inbound_response.json()['json']

        assert inbound_json['card_cvc'] != self.data['card_cvc']
        assert inbound_json['card_number'] != self.data['card_number']

        assert re.findall(r'tok_sat_[\w\d]+', inbound_json['card_cvc'])
        assert re.findall(r'^4[0-9]{12}(?:[0-9]{3})?$', inbound_json['card_number'])

        proxies = {
            'http': f'{self.satellite_url}:{self.forward_proxy_port}',
            'https': f'{self.satellite_url}:{self.forward_proxy_port}'
        }
        outbound_response = requests.post('https://echo.apps.verygood.systems/post',
                                 headers=self.headers,
                                 json=inbound_json,
                                 proxies=proxies,
                                 verify=False)
        outbound_json = outbound_response.json()['json']

        assert outbound_json['card_cvc'] == self.data['card_cvc']
        assert outbound_json['card_number'] == self.data['card_number']

To keep everything consistent, I will wrap our tests in another docker container. For that, we need to create a requirements.txt file:

pytest == 6.2.2
requests == 2.25.1

Add a dockerfile for building our tests image:

FROM python:3.9.1-slim-buster

WORKDIR /app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . /app

CMD ["pytest", "./route_tests.py"]

As a final touch, add the following code snippet to our docker-compose.yaml:

vgs-tests:
  build:
    context: .
    dockerfile: Dockerfile
  volumes:
    - .:/app/
  environment:
    VGS_SATELLITE_URL: http://vgs-satellite

Now, we can run our tests simply by invoking

$ docker-compose run vgs-tests

Creating git-cli-satellite-example_vgs-tests_run ... done
==================================== test session starts ==========================
platform linux -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /app
collected 1 item

route_tests.py .                                                             [100%]

===================================== 1 passed in 1.41s ===========================

If everything ran successfully, we are finally ready to apply our updated routes to our VGS vault.

Applying the changes to VGS vault

To apply the newly created routes to our vault, make sure that the VGS Cli is running and authenticated. Then, use the vgs apply command:

$ docker-compose up -d vgs-cli

Starting git-cli-satellite-example_vgs-cli_1 ... done

$ docker-compose exec vgs-cli vgs login

Can not open your browser, please open this URL in your browser: https://auth.verygoodsecurity.com/....

$ docker-compose exec vgs-cli vgs apply routes --vault tntfpb2cmsw -f /data/vgs_routes.yaml

Route 4a74170c-2638-45c8-8880-1ea3cf66f38a processed
Route 6d2135c7-f20b-46ba-bee6-e58159c42e6d processed
Routes updated successfully for vault tntfpb2cmsw

Now, we can send a couple of request to our vault, and check the access logs to see if everything worked correctly:

$ docker-compose exec vgs-cli vgs logs access --vault tntfpb2cmsw --tail 10

data:
- attributes:
    data_environment: sandbox
    expired_at: null
    http:
      method: POST
    occurred_at: '2021-02-18T12:50:56.062000+00:00'
    path: https://echo.apps.verygood.systems:443/post
    pinned: false
    protocol: http
    proxy_status: '200'
    route_type: reverse
    routes:
      data:
      - attributes:
          matched: true
          route_id: 4a74170c-2638-45c8-8880-1ea3cf66f38a
          trace_id: 2373598dce4f37d3e0c77aa7247b5f51
        id: 70ffd363-88fe-4d55-9fee-1c7b1318e94d
        type: routes
    source: ''
    tenant_id: tntfpb2cmsw
    trace_id: 2373598dce4f37d3e0c77aa7247b5f51
    upstream: echo.apps.verygood.systems:443
    upstream_status: '200'
    upstream_time_ms: 13
    user_id: ''
  id: 2373598dce4f37d3e0c77aa7247b5f51
  links:
    self: /access-logs/2373598dce4f37d3e0c77aa7247b5f51
  type: access-log
- attributes:
    data_environment: sandbox
    expired_at: null
    http:
      method: POST
    occurred_at: '2021-02-18T12:52:33.750000+00:00'
    path: https://echo.apps.verygood.systems:443/post
    pinned: false
    protocol: http
    proxy_status: '200'
    route_type: forward
    routes:
      data:
      - attributes:
          matched: true
          route_id: 6d2135c7-f20b-46ba-bee6-e58159c42e6d
          trace_id: 37cb2fb6999d1012314592d696718d97
        id: e751936f-af14-45af-be70-13e7abc3d36b
        type: routes
    source: ''
    tenant_id: tntfpb2cmsw
    trace_id: 37cb2fb6999d1012314592d696718d97
    upstream: echo.apps.verygood.systems:443
    upstream_status: '200'
    upstream_time_ms: 5
    user_id: USx6vrWo4tKMwPy25r3hYwfn
  id: 37cb2fb6999d1012314592d696718d97
  links:
    self: /access-logs/37cb2fb6999d1012314592d696718d97
  type: access-log
version: 1

VGS CLI Login Automation

If you’d like to remove the need to run vgs login and authenticate every time you work with the CLI, there’s a way to enable auto login using client credentials. Once you have received your credentials, all you have to do is add those to your docker-compose.yaml file:

vgs-cli:
  image: quay.io/verygoodsecurity/vgs-cli:${VERSION:-latest}
  stdin_open: true # docker run -i
  tty: true # docker run -t
  volumes:
    - ./:/data/
  # environment:
  # VGS_CLIENT_ID: example
  # VGS_CLIENT_SECRET: example
  command: bash
  ports:
    - '7745:7745'
    - '8390:8390'
    - '9056:9056'

Full code for these examples can be found on our github.

References

• VGS: https://www.verygoodsecurity.com/about-us