Selection functions
lookup
Commentary
added in 0.0.5
Retrieves a previously generated event from the supplied collection, optionally drilling into it with a path. This lets you create relationships, through shared identifiers, across different data sets.
ShadowTraffic guarantees that events will only be available for lookup after they have been successfully written to the target collection.
When looking up data, you can either target a generator by its output target 1 or its generator name. 2
By default, the event chosen from a lookup is entirely random. But if you like, you can override how the event is chosen by targeting the first generated event 3, the last generated event 4, or a histogram to bias the population selection. 5
In addition to looking up any piece of generated data, you can also access its variables at the time of generation. 6
Examples
Kafka lookup
Look up data in another Kafka topic.
{
"generators": [
{
"topic": "a",
"key": {
"id": {
"_gen": "string",
"expr": "#{Name.firstName}"
}
}
},
{
"topic": "b",
"value": {
"_gen": "lookup",
"topic": "a",
"path": [
"key",
"id"
]
}
}
],
"connections": {
"kafka": {
"kind": "kafka",
"producerConfigs": {
"bootstrap.servers": "localhost:9092",
"key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
}
}
}
}
[
{
"topic": "a",
"key": {
"id": "Moises"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Moises",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Buddy"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Buddy",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Patricia"
},
"value": null,
"headers": null
}
]
Postgres lookup
Look up data in a Postgres table.
{
"generators": [
{
"table": "a",
"row": {
"id": {
"_gen": "string",
"expr": "#{Name.firstName}"
}
}
},
{
"table": "b",
"row": {
"id": {
"_gen": "lookup",
"table": "a",
"path": [
"row",
"id"
]
}
}
}
],
"connections": {
"postgres": {
"kind": "postgres",
"connectionConfigs": {
"host": "localhost",
"port": 5432,
"username": "postgres",
"password": "postgres",
"db": "mydb"
}
}
}
}
[
{
"table": "a",
"row": {
"id": "Merlin"
},
"op": null,
"where": null
},
{
"table": "b",
"row": {
"id": "Merlin"
},
"op": null,
"where": null
},
{
"table": "a",
"row": {
"id": "Chauncey"
},
"op": null,
"where": null
},
{
"table": "b",
"row": {
"id": "Merlin"
},
"op": null,
"where": null
},
{
"table": "a",
"row": {
"id": "Jesus"
},
"op": null,
"where": null
}
]
Periodic lookups
Sometimes make a new key, sometimes use a previously generated one.
{
"generators": [
{
"topic": "users",
"key": {
"_gen": "weightedOneOf",
"choices": [
{
"weight": 5,
"value": {
"_gen": "string",
"expr": "#{Name.fullName}"
}
},
{
"weight": 5,
"value": {
"_gen": "lookup",
"topic": "users",
"path": [
"key"
]
}
}
]
}
}
],
"connections": {
"kafka": {
"kind": "kafka",
"producerConfigs": {
"bootstrap.servers": "localhost:9092",
"key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
}
}
}
}
[
{
"topic": "users",
"key": "Vita Adams DVM",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Cortney Emmerich",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Cortney Emmerich",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Efren Gleason III",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Ms. Eddy Medhurst",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Efren Gleason III",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Harold Cartwright",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Vita Adams DVM",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Cortney Emmerich",
"value": null,
"headers": null
},
{
"topic": "users",
"key": "Isiah Johns",
"value": null,
"headers": null
}
]
Explicit connections
Explicitly supply the connection name when there are multiple connections.
{
"generators": [
{
"connection": "postgres",
"table": "a",
"row": {
"email": {
"_gen": "string",
"expr": "#{Internet.emailAddress}"
}
}
},
{
"connection": "kafka",
"topic": "b",
"value": {
"_gen": "lookup",
"connection": "postgres",
"table": "a",
"path": [
"row",
"email"
]
}
}
],
"connections": {
"kafka": {
"kind": "kafka",
"producerConfigs": {
"bootstrap.servers": "localhost:9092",
"key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
}
},
"postgres": {
"kind": "postgres",
"connectionConfigs": {
"host": "localhost",
"port": 5432,
"username": "postgres",
"password": "postgres",
"db": "mydb"
}
}
}
}
[
{
"table": "a",
"row": {
"email": "darrin.hansen@hotmail.com"
},
"op": null,
"where": null
},
{
"topic": "b",
"key": null,
"value": "darrin.hansen@hotmail.com",
"headers": null
},
{
"table": "a",
"row": {
"email": "syble.mckenzie@yahoo.com"
},
"op": null,
"where": null
},
{
"topic": "b",
"key": null,
"value": "syble.mckenzie@yahoo.com",
"headers": null
},
{
"table": "a",
"row": {
"email": "corinne.wiegand@yahoo.com"
},
"op": null,
"where": null
}
]
Targeting the first event
Set strategy to first to lookup the first event generated. This is useful if you want to grab the data a generator initialized with. Note tthat "first" refers to the head of the generator's history, so as its history deletes data in a first-in-first out manner, this value may change.
In this example, notice how all the values for b lock onto the first value generated by a.
[
{
"topic": "a",
"value": {
"_gen": "normalDistribution",
"mean": 100,
"sd": 10
}
},
{
"topic": "b",
"value": {
"_gen": "lookup",
"strategy": "first",
"topic": "a",
"path": [
"value"
]
}
}
]
[
{
"topic": "a",
"key": null,
"value": 103.14600491911256,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 103.14600491911256,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 98.85385285252646,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 103.14600491911256,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 86.9507120054815,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 103.14600491911256,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 90.5285596171052,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 103.14600491911256,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 102.13496193317972,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 103.14600491911256,
"headers": null
}
]
Targeting the last event
Set strategy to last to lookup the latest event generated.
In this example, notice how each value for b lock onto the most revent value generated by a.
[
{
"topic": "a",
"value": {
"_gen": "normalDistribution",
"mean": 100,
"sd": 10
}
},
{
"topic": "b",
"value": {
"_gen": "lookup",
"strategy": "last",
"topic": "a",
"path": [
"value"
]
}
}
]
[
{
"topic": "a",
"key": null,
"value": 113.87710255682987,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 113.87710255682987,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 105.28750060902658,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 105.28750060902658,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 74.51186570054317,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 74.51186570054317,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 94.25258810408184,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 94.25258810408184,
"headers": null
},
{
"topic": "a",
"key": null,
"value": 92.88628994941408,
"headers": null
},
{
"topic": "b",
"key": null,
"value": 92.88628994941408,
"headers": null
}
]
Controlling distribution
Use a histogram to control how the element is selected from the population. This invocation chooses 20% of the elements 80% of the time from a Kafka topic.
{
"generators": [
{
"topic": "a",
"key": {
"id": {
"_gen": "string",
"expr": "#{Name.fullName}"
}
}
},
{
"topic": "b",
"value": {
"_gen": "lookup",
"topic": "a",
"path": [
"key",
"id"
],
"histogram": {
"_gen": "histogram",
"bins": [
{
"bin": 0.2,
"frequency": 8
},
{
"bin": 0.8,
"frequency": 2
}
]
}
}
}
],
"connections": {
"kafka": {
"kind": "kafka",
"producerConfigs": {
"bootstrap.servers": "localhost:9092",
"key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
}
}
}
}
[
{
"topic": "a",
"key": {
"id": "Magaly Boyle"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Magaly Boyle",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Tyree O'Keefe"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Magaly Boyle",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Wilson Breitenberg"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Magaly Boyle",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Germaine Donnelly"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Magaly Boyle",
"headers": null
},
{
"topic": "a",
"key": {
"id": "Allyson Mraz"
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": "Magaly Boyle",
"headers": null
}
]
Multiple lookups
If you need to look up multiple fields, be careful not to use lookup more than once in the same generator. Multiple calls are not consistent. In other words, they won't return the same event in each call.
Instead, call lookup just once by using a variable and setting path to [] to grab then entire event. Then pick out the relevant fields.
[
{
"topic": "a",
"key": {
"name": {
"_gen": "string",
"expr": "#{Name.fullName}"
},
"magicNumber": {
"_gen": "uniformDistribution",
"bounds": [
0,
100
],
"decimals": 0
}
}
},
{
"topic": "b",
"vars": {
"result": {
"_gen": "lookup",
"topic": "a",
"path": []
}
},
"value": {
"lookedUpName": {
"_gen": "var",
"var": "result",
"path": [
"key",
"name"
]
},
"lookedUpNumber": {
"_gen": "var",
"var": "result",
"path": [
"key",
"magicNumber"
]
}
}
}
]
[
{
"topic": "a",
"key": {
"name": "Bernie Waelchi",
"magicNumber": 65
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"lookedUpName": "Bernie Waelchi",
"lookedUpNumber": 65
},
"headers": null
},
{
"topic": "a",
"key": {
"name": "Edgardo Heaney",
"magicNumber": 87
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"lookedUpName": "Bernie Waelchi",
"lookedUpNumber": 65
},
"headers": null
},
{
"topic": "a",
"key": {
"name": "Matilda Fritsch DDS",
"magicNumber": 46
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"lookedUpName": "Edgardo Heaney",
"lookedUpNumber": 87
},
"headers": null
},
{
"topic": "a",
"key": {
"name": "Miss Roland Schaden",
"magicNumber": 72
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"lookedUpName": "Edgardo Heaney",
"lookedUpNumber": 87
},
"headers": null
},
{
"topic": "a",
"key": {
"name": "Noel Streich",
"magicNumber": 68
},
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"lookedUpName": "Bernie Waelchi",
"lookedUpNumber": 65
},
"headers": null
}
]
Lookup by name
Use name in the lookup function to target another generator by its name attribute. This is useful if multiple generators write to the same output connection and collection and you want to distinguish between them.
[
{
"name": "generator-A",
"topic": "a",
"key": {
"_gen": "uniformDistribution",
"bounds": [
0,
100
],
"decimals": 0
}
},
{
"topic": "b",
"value": {
"result": {
"_gen": "lookup",
"name": "generator-A",
"path": [
"key"
]
}
}
}
]
[
{
"topic": "a",
"key": 45,
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"result": 45
},
"headers": null
},
{
"topic": "a",
"key": 76,
"value": null,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"result": 76
},
"headers": null
},
{
"topic": "a",
"key": 72,
"value": null,
"headers": null
}
]
Accessing variables
In addition to event data, you may want to access the variables that were set when the looked-up event was created. Simply use vars as the first key in path (you can of course drill in further).
vars will contain all variables by name set from both varsOnce and `vars.
In the example below, a generates the sum of two numbers. b looks up the numbers that were added together, even though they aren't visible in the final output of a. This example uses strategy set to last so you can easily inspect the output between a and b.
[
{
"topic": "a",
"varsOnce": {
"x": {
"_gen": "normalDistribution",
"mean": 50,
"sd": 5
}
},
"vars": {
"y": {
"_gen": "uniformDistribution",
"bounds": [
0,
100
]
}
},
"value": {
"_gen": "math",
"expr": "x + y"
}
},
{
"topic": "b",
"value": {
"result": {
"_gen": "lookup",
"topic": "a",
"path": [
"vars"
],
"strategy": "last"
}
}
}
]
[
{
"topic": "a",
"key": null,
"value": 138.03133666390528,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"result": {
"x": 46.598322466188684,
"y": 91.43301419771659
}
},
"headers": null
},
{
"topic": "a",
"key": null,
"value": 56.654100485557365,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"result": {
"x": 46.598322466188684,
"y": 10.055778019368677
}
},
"headers": null
},
{
"topic": "a",
"key": null,
"value": 90.3780812617409,
"headers": null
},
{
"topic": "b",
"key": null,
"value": {
"result": {
"x": 46.598322466188684,
"y": 43.77975879555221
}
},
"headers": null
}
]
Specification
JSON schema
Lookups against different connection types have different schemas. Each schema is listed below in array form.
[
{
"name": "Kafka",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"topic": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"topic",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "Postgres",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"table": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"table",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "sqlServer",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"table": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"table",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "Proton",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"stream": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"stream",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "Timeplus",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"stream": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"stream",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "S3",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"bucket": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"keyPrefix": {
"type": "string"
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"bucket",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "googleCloudStorage",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"bucket": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"blobPrefix": {
"type": "string"
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"bucket",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "azureBlobStorage",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"container": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"keyPrefix": {
"type": "string"
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"container",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "fileSystem",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"directory": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"directory",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "EventStore",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"stream": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"stream",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
},
{
"name": "Webhook",
"schema": {
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"url": {
"type": "string"
},
"strategy": {
"type": "string",
"enum": [
"first",
"last",
"random"
]
},
"name": {
"type": "string"
},
"path": {
"type": "array",
"items": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "string"
}
]
}
}
},
"oneOf": [
{
"required": [
"url",
"path"
]
},
{
"required": [
"name",
"path"
]
}
]
}
}
]