Cheat sheet

Use these commands when you need a bit of boilerplate to get up and running.

Docker commands

Container start

docker run --env-file license.env \
           -v $(pwd)/your-config.json:/home/config.json \
           shadowtraffic/shadowtraffic:latest \
           --config /home/config.json \
           --watch --sample 10 --stdout

• license.env contains your license environment variables and values.
• your-config.json is your ShadowTraffic config file.

Docker Compose example

shadowtraffic:
  image: shadowtraffic/shadowtraffic:latest
  volumes:
    - .:/workspace
  entrypoint: ["java", "-Djava.security.manager=allow", "-jar", "/home/shadowtraffic.jar"]
  command: ["--config", "/workspace/your-config.json"]
  env_file:
    - shadowtraffic/license.env

---

The CLI

CLI switches

      --action <action>                          What ShadowTraffic should do (run, bootstrap). Defaults to run.
      --bootstrap-from-avro-schema <file>        Uses the provided Avro Schema file to approximate the ShadowTraffic configuration.
      --bootstrap-from-json-schema <file>        Uses the provided JSON Schema file to approximate the ShadowTraffic configuration.
      --bootstrap-to <to>                        The target connection type to bootstrap the ShadowTraffic configuration file to.
  -c, --config <file>                            Path to configuration file.
      --config-base64 <text>                     Instead of a file, use this Base64 encoded configuration data.
      --config-format <format>             json  Format of the config file: must be one of (json, yaml). Default is json.
  -h, --help                                     Display this information.
      --no-pretty                                Do not use a pretty printer with --stdout when generating data.
  -q, --quiet                                    Do not print any status text other than generated data to the command line.
      --report-benchmark                         Prints performance statistics after all generators complete.
      --sample <n>                               Generates the specified number of events and then stops immediately.
      --seed <n>                                 Initializes random generators with this seed, enabling repeatable runs.
  -s, --stdout                                   Ignore original output targets and forward all generated data to standard out.
  -w, --watch                                    Keep running and restart generation on config file changes.
      --with-studio                              Starts ShadowTraffic Studio on port 8080 to visually interact with your generated data in the browser. Must run with --watch and --sample.

---

Metrics

Prometheus metrics

ShadowTraffic exposes metrics through Prometheus. Inside the container, curl http://localhost:9400 to see the metrics, and forward port 9400 to your host if you want to ingest or graph the metrics into your own observability platform.

All metrics use two labels - connection name, and output target. For example, if you're connection is to Kafka and it's named "kafka", and the topic you're writing to is named "clicks", you'll see a label ("kafka", "clicks").

Metric list

name	kind	description
generator_events_sent_total	counter	The total number of events successfully written to the target system. Use this counter with Prometheus rate to calculate events/second.
generator_iteration_duration_seconds	histogram	Measures the duration it takes to generate a single event, excluding the time to write it to your target system. Use this to see how fast or slow the data creation part of your ShadowTraffic configuration is.

All standard JVM metrics are also sent through Prometheus's JvmMetrics reporter.

Bootstrap

Bootstrapping

If you already have a formal schema that your data needs to adhere to, you can use ShadowTraffic's bootstrap functionality to save you some work. By specifying the schema and a connection type, ShadowTraffic will convert your schema into a configuration file, stubbing out values where appropriate. This can save you a ton of typing if your schema is large.

Currently, JSON Schema and Avro are supported as an input source. See below.

JSON schema

As an example, imagine you have a simple JSON schema file named mySchema.json:

{
    "type": "object",
    "properties": {
        "id": { "type": "integer" },
        "name": { "type": "string" },
        "age": { "type": "integer" },
        "isActive": { "type": "boolean" },
    },
    "required": [ "id", "name" ]
}

You can use bootstrap to turn this schema into a configuration file with stubs for each concrete value. In the following command, mySchema.json is mounted into the container so that ShadowTraffic can see it.

docker run --env-file license.env -v $(pwd)/mySchema.json:/home/mySchema.json shadowtraffic/shadowtraffic:latest --action bootstrap --bootstrap-from-json-schema /home/mySchema.json --bootstrap-to kafka

The switch --action bootstrap tells ShadowTraffic to try and write a configuration file. Alongside are the --bootstrap-from-json-schema and --bootstrap-to switches, the latter of which indicates which connection type the data should be generated to.

After running this command, ShadowTraffic should output roughly the following stub with instructions on how to run it. 🎉

{
  "generators" : [
    {
      "topic" : "sandbox",
      "value" : {
        "id" : {
          "_gen" : "uniformDistribution",
          "bounds" : [
            1,
            50
          ],
          "decimals" : 0
        },
        "name" : {
          "_gen" : "string",
          "expr" : "#{Name.fullName}"
        },
        "age" : {
          "_gen" : "uniformDistribution",
          "bounds" : [
            1,
            50
          ],
          "decimals" : 0
        },
        "isActive" : {
          "_gen" : "boolean"
        }
      }
    }
  ],
  "connections" : {
    "kafka" : {
      "kind" : "kafka",
      "producerConfigs" : {
        "bootstrap.servers" : "localhost9092",
        "key.serializer" : "io.shadowtraffic.kafka.serdes.JsonSerializer",
        "value.serializer" : "io.shadowtraffic.kafka.serdes.JsonSerializer"
      }
    }
  }
}

Avro

Just like bootstrapping from a JSON schema, you can bootstrap from an Avro schema, too. Use the following command, with the steps the same as JSON scheama:

docker run --env-file license.env -v $(pwd)/mySchema.avsc:/home/mySchema.avsc shadowtraffic/shadowtraffic:latest --action bootstrap --bootstrap-from-avro-schema /home/mySchema.avsc --bootstrap-to kafka

Image locations

ECR

One of the downsides of storing the ShadowTraffic image in DockerHub is that pulling is subject to rate limits, which can be a problem if you don't have a paid plan. To work around that, you also pull the image from Amazon Elastic Container Registry, where it's listed as a public image with far more generous pull limits:

docker pull public.ecr.aws/s2b8p3j6/shadowtraffic/shadowtraffic:$version

Operations

Debugging the container

If the ShadowTraffic container isn't running how you expect it to, set the environment variable to SHADOWTRAFFIC_DEBUG to any value to force ShadowTraffic into debug mode.

In debug mode, ShadowTraffic will complete its run by printing:

1. Available system resources, like allocated CPUs and max memory
2. The Java version its running on
3. The command line arguments it received at startup
4. The environment variables it can see
5. The Java properties it was set with

Additionally, ShadowTraffic will indefinitely suspend its process to stay running. This makes it easier to log into the container and further investigating you might want to do.