loadPropertiesFile

Commentary

added in 2.1.0

Special pre-processor function that loads Java properties files. This is most useful for connecting to Java-based backends. Kafka is a good example of this, where your broker credentials probably already live in a client.properties file that the rest of your tooling reads.

This function can be used in one of two ways. First, you can use the file parameter to load a single file in place. ¹ Second, you can use the files parameter to load a group of files and merge them together, with later files taking priority. ²

In either case, the paths listed must be fully-qualified to files inside the ShadowTraffic Docker container. Be sure to volume mount these files in addition to your base configuration.

In some cases, you may need to add extra fields that aren't present in your properties file. Here again, Kafka is a good example. A typical client.properties doesn't contain required key.serializer and value.serializer fields. To add more fields, use the optional overrides key. Its contents are merged into whatever the file contained, so it both adds missing keys and takes precedence on any key the file already set.

You can also supply a data parameter to do simple primitive substitutions according to the Mustache spec. This is provided as a convenience. If you need something more sophisticated, use something like Jinja directly.

Note that pre-processing functions, like this one, don't evaluate function modifiers because they execute only once before runtime. So keys like elide and null are ignored.

---

Caveats

Untyped values

Remember that in Properties files, every value is loaded as a string—exactly as Java would read it, including values that look like numbers or booleans. So a file containing:

linger.ms=100
enable.idempotence=true

loads as:

{
    "linger.ms": "100",
    "enable.idempotence": "true"
}

This is mostly what Java-based backends expect. So while it is worth knowing, it's rarely a problem.

Container paths

If your properties file points at other files on disk (Java keystore and truststore, for example) those paths are passed through untouched. They must resolve inside the container, which means volume mounting the keystores as well and writing container paths in the properties file itself.

ssl.keystore.location=/etc/shadowtraffic/client.keystore.p12
ssl.truststore.location=/etc/shadowtraffic/client.truststore.jks

Static data

Note that the contents of data must either be static data or further preprocessor functions. Runtime functions won't be applied before data is passed into the loaded file, for the same reason described in loadJsonFile.

---

Examples

Loading a file

If client.properties contained:

bootstrap.servers=pkc-abc.us-west4.gcp.confluent.cloud:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='KEY' password='SECRET';

Specify the file with a fully qualified path, and use overrides to add the serializers, which this properties file doesn't contain:

{
    "kind": "kafka",
    "producerConfigs": {
        "_gen": "loadPropertiesFile",
        "file": "/path/to/client.properties",
        "overrides": {
            "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
            "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
        }
    }
}

The resulting connection would be equivalent to writing all of those keys inline, plus the two serializers from overrides.

When you load properties files, you should see messages like the following on the console:

✝ ***
✝ Replacing properties file /path/to/client.properties at path [ "connections", "kafka", "producerConfigs" ]
✝ ***

Merging files

Use files to merge multiple files together, with later files taking priority over earlier ones. This is useful for keeping shared connection settings in one file and per-environment settings in another.

For example, if base.properties contained:

bootstrap.servers=localhost:9092
acks=1

And prod.properties contained:

acks=all
linger.ms=100

And you used the following load call:

{
    "kind": "kafka",
    "producerConfigs": {
        "_gen": "loadPropertiesFile",
        "files": [
            "/path/to/base.properties",
            "/path/to/prod.properties"
        ],
        "overrides": {
            "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
            "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
        }
    }
}

The result would be:

{
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "linger.ms": "100",
    "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
    "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
}

When you merge properties files, you should see messages like the following on the console:

✝ ***
✝ Merging properties files [/path/to/base.properties, /path/to/prod.properties] at path [ "connections", "kafka", "producerConfigs" ]
✝ ***

Substitute fields

Use data to substitute primitives into the target file where names are denoted with {{ name }} expressions. Because all other preprocessors run first, you can feed an environment variable in as data.

Here, client.properties contains {{ env }}, which is replaced by the value of the DEPLOY_ENV environment variable before the file is parsed.

{
    "kind": "kafka",
    "producerConfigs": {
        "_gen": "loadPropertiesFile",
        "file": "/path/to/client.properties",
        "data": {
            "env": {
                "_gen": "env",
                "var": "DEPLOY_ENV"
            }
        },
        "overrides": {
            "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
            "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
        }
    }
}

---

Specification

JSON schema

{
    "type": "object",
    "properties": {
        "file": {
            "type": "string"
        },
        "files": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "data": {
            "type": "object"
        },
        "overrides": {
            "type": "object"
        }
    },
    "oneOf": [
        {
            "required": [
                "file"
            ]
        },
        {
            "required": [
                "files"
            ]
        }
    ]
}