lookupScope

Commentary

added in 0.18.0

global configuration

lookupScope controls the timing with which events can be captured in a lookup function.

By default, as soon as an event is generated, it becomes available to be referenced by lookup. This is generally the behavior that's most intuitive and helpful.

But some of ShadowTraffic's connectors, like Kafka and S3, send events asynchronously to increase write performance. In these cases, you may want to prohit events from being looked up until they have been successfully committed to the target system—whenever that happens. ¹

Note that when you set committed scope, you will likely see divergent behavior between --stdout and producing data against a real system. This happens because --stdout prints events synchronously, whereas your chosen connector might produce events asynchronously. The difference in timing is what causes the change in behavior.

Most commonly, you'll at the start of a run, you'll see your generator with the lookup call not run for a bit. This is because its waiting for data to be committed on the referenced generators.

Finally, note that this configuration can only be set at the global level.

---

Examples

Setting committed scope

Use globalConfigs to set lookupScope to committed. This will prevent events from being looked up until they have been fully committed to their target backends.

{
    "generators": [
        {
            "topic": "topic-a",
            "value": {
                "_gen": "boolean"
            }
        },
        {
            "topic": "topic-b",
            "value": {
                "_gen": "lookup",
                "topic": "topic-a",
                "path": [
                    "value"
                ]
            }
        }
    ],
    "globalConfigs": {
        "lookupScope": "committed"
    },
    "connections": {
        "kafka": {
            "kind": "kafka",
            "producerConfigs": {
                "bootstrap.servers": "localhost:9092",
                "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
                "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
            }
        }
    }
}

Setting instantaneous scope

Use globalConfigs to set explicitly lookupScope to instantaneous. This is the default, so ommitting this configuration is the same as setting it to this value.

{
    "generators": [
        {
            "topic": "topic-c",
            "value": {
                "_gen": "uuid"
            }
        },
        {
            "topic": "topic-d",
            "value": {
                "_gen": "lookup",
                "topic": "topic-c",
                "path": [
                    "value"
                ]
            }
        }
    ],
    "globalConfigs": {
        "lookupScope": "instantaneous"
    },
    "connections": {
        "kafka": {
            "kind": "kafka",
            "producerConfigs": {
                "bootstrap.servers": "localhost:9092",
                "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
                "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer"
            }
        }
    }
}

---

Specification

JSON schema

{
    "type": "string",
    "enum": [
        "instantaneous",
        "committed"
    ]
}