Profiler

Commentary

Evolving feature: subject to change

Many times when you need to create synthetic data, you're working backwards from a hypothesis. For example, you may want to demonstrate what happens in your project when it encounters some critical trait or edge case in your data, so you carefully configure ShadowTraffic with a particular schema, data range, and the like.

But other times, you might want to simulate an existing data source—one you may not know much about, or is so large and complex that it would be tedious to simulate by hand.

To that end, ShadowTraffic Profiler is a built-in tool that connects to existing data sources and automatically generates a matching ShadowTraffic simulation. It statistically models your schema and data, determining a representative simulation that retains its essential properties without compromising privacy of the original data.

Motivation

Some examples that motivate where you'd use this:

1. You have an existing data source and want a zero-configuration way to make more data like it.
2. You want to simulate hundreds or thousands of tables/topics.
3. Your schemas are complex, with many columns and deep nesting.
4. The properties of your data aren't obvious or easily discoverable, like the distribution of numeric values.
5. The relationships (foreign keys) between your data sources aren't obvious.
6. Your data contains outliers and schema violations, and you want to simulate those in similar proportion.

How it works

Configuring the profiler

Like all other features of ShadowTraffic, the Profiler launches with the help of a few command line arguments. In this case, the Profiler needs to know only one thing: where are your existing data sources? --config supplies that information:

docker run --env-file license.env \
           -v $(pwd)/your-config.json:/home/config.json \
           shadowtraffic/shadowtraffic:latest \
           --action profile \
           --config /home/config.json

The content of config.json largely mirrors a standard ShadowTraffic configuration file, except you only need to supply a connections block. For example, to profile data from Kafka, a basic config.json may look like:

{
    "connections": {
        "prod-kafka": {
            "kind": "kafka",
            "consumerConfigs": {
                "bootstrap.servers": "broker:9092",
                "schema.registry.url": "http://schema-registry:8081",
                "key.deserializer": "org.apache.kafka.common.serialization.StringDeserializer",
                "value.deserializer": "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            },
            "topics": [
                {
                    "topic": "orders"
                }
            ]                
        }
    }
}

With that, the Profiler will boot up and use everything it can to learn what the corresponding complete ShadowTraffic file should look like. In this case, it might output something like:

Example output

{
    "generators" : [
        {
            "connection" : "prod-kafka",
            "topic" : "orders",
            "value" : {
                "order_id" : {
                    "_gen" : "uuid"
                },
                "status" : {
                    "_gen" : "oneOf",
                    "choices" : [
                        "PENDING",
                        "SHIPPED",
                        "DELIVERED",
                        "CANCELLED"
                    ]
                },
                "amount" : {
                    "_gen" : "normalDistribution",
                    "mean" : 100,
                    "sd" : 10
                },
                "placed_at" : {
                    "_gen" : "uniformDistribution",
                    "bounds" : [
                        1,
                        5000000
                    ],
                    "decimals" : 0
                },
                "shipped_at" : {
                    "_gen" : "oneOf",
                    "choices" : [
                        {
                            "null" : null
                        },
                        {
                            "long" : {
                                "_gen" : "uniformDistribution",
                                "bounds" : [
                                    1,
                                    5000000
                                ],
                                "decimals" : 0
                            }
                        }
                    ]
                },
                "customer" : {
                    "name" : {
                        "_gen" : "string",
                        "expr" : "#{Name.fullName}"
                    }
                },
                "tags" : {
                    "_gen" : "repeatedly",
                    "n" : {
                        "_gen" : "uniformDistribution",
                        "bounds" : [
                            1,
                            5
                        ],
                        "decimals" : 0
                    },
                    "target" : {
                        "_gen" : "string",
                        "expr" : "#{Cosmere.metals}"
                    }
                },
                "note" : {
                    "_gen" : "oneOf",
                    "choices" : [
                        {
                            "null" : null
                        },
                        {
                            "string" : {
                                "_gen" : "string",
                                "expr" : "#{Yoda.quote}"
                            }
                        }
                    ]
                }
            },
            "producerConfigs" : {
                "value.serializer" : "io.confluent.kafka.serializers.KafkaAvroSerializer"
            },
            "key" : null
        }
    ],
    "connections" : {
        "prod-kafka" : {
            "kind" : "kafka",
            "producerConfigs" : {
                "bootstrap.servers" : "broker:9092",
                "schema.registry.url" : "http://schema-registry:8081",
                "key.serializer" : "org.apache.kafka.common.serialization.StringSerializer",
                "value.serializer" : "org.apache.kafka.common.serialization.StringSerializer"
            }
        }
    }
}

Profiling is currently only supported for certain connections. For programmatic usage, it's recommended to invoke profiling with the --quiet flag to elide logging output.

Examples

Profiling Kafka

Profiling a Kafka cluster requires two parameters:

1. consumerConfigs: your broker endpoint and Schema Registry endpoint to connect
2. topics: an array of topics to simulate

Currently, ShadowTraffic will only profile topics whose key or value are linked to Schema Registry. It assumes a default TopicNameStrategy naming convention, but both the key and value subjects can be overridden, as seen in the example.

An example config.json file might look like:

{
    "connections": {
        "prod-kafka": {
            "kind": "kafka",
            "consumerConfigs": {
                "bootstrap.servers": "broker:9092",
                "schema.registry.url": "http://schema-registry:8081",
                "key.deserializer": "io.confluent.kafka.serializers.KafkaAvroDeserializer",
                "value.deserializer": "io.confluent.kafka.serializers.KafkaAvroDeserializer"
            },
            "topics": [
                {
                    "topic": "topic-a"
                },
                {
                    "topic": "topic-b",
                    "keySubjectName": "specialKeySubject",
                    "valueSubjectName": "nonstandardValueSubject"
                }
            ]
        }
    }
}