Connections
filesystem
Commentary
added in 0.5.10
Writes to the local file system.
Specify the path to write to in the connection map. If any of the parent directories don't exist, they will automatically be created. 1
ShadowTraffic will write a series of files to that path, periodically rolling a new one. Specify the base file name with fileName, and files will be generated in the pattern fileName-n.suffix. The value of n will always increase on the latest file generated. 2
By default, a new file is created every 500 ms or 5000 elements, whichever happens first. You can override object size/write timing with batchConfigs. 3
Do not write multiple generators that output to the same file name in the same directory, since the output writers will contend with each other and overwrite previously written content.
Formats
Set the data format with the fileConfig parameter. json and jsonl are currently supported.
Examples
Configuring the connection
At minimum, use path specify to where the files should be written.
{
"connections": {
"localFs": {
"kind": "fileSystem",
"path": "/tmp/data"
}
}
}
Writing to files
Use fileName to set the base file name to be written. In this example, files named transactions-0.json, transactions-n.json, ... will be created. Use data to set the data to be written to the file.
{
"fileName": "transactions",
"fileConfigs": {
"format": "json"
},
"data": {
"amount": {
"_gen": "normalDistribution",
"mean": 100,
"sd": 2
}
},
"connections": {
"localFs": {
"kind": "fileSystem",
"path": "/tmp/data"
}
}
}
Setting batch size
Use batchConfigs to control how frequently files are rolled. lingerMs will roll a new file after the specified duration has passed. batchElements will roll a new file when the specified number of events have been generated. A new file be rolled for whichever condition comes first.
{
"fileName": "secretNumbers",
"fileConfigs": {
"format": "json"
},
"data": {
"n": {
"_gen": "uniformDistribution",
"bounds": [
1,
1000
],
"decimals": 0
}
},
"connections": {
"localFs": {
"kind": "fileSystem",
"path": "/tmp/data",
"batchConfigs": {
"lingerMs": 5000,
"batchElements": 50000
}
}
}
}
Pretty printing
Use pretty to pretty print the contents in the file.
{
"fileName": "ipAddresses",
"fileConfigs": {
"format": "json",
"pretty": true
},
"data": {
"ip": {
"_gen": "string",
"expr": "#{Internet.ipV4Address}"
},
"timestamp": {
"_now": "now"
}
},
"connections": {
"localFs": {
"kind": "fileSystem",
"path": "/tmp/data"
}
}
}
Specification
Connection JSON schema
{
"type": "object",
"properties": {
"kind": {
"type": "string",
"const": "fileSystem"
},
"path": {
"type": "string"
},
"batchConfigs": {
"type": "object",
"properties": {
"lingerMs": {
"type": "integer",
"minimum": 0
},
"batchElements": {
"type": "integer",
"minimum": 1
}
}
}
},
"required": [
"path"
]
}
Generator JSON schema
{
"type": "object",
"properties": {
"connection": {
"type": "string"
},
"name": {
"type": "string"
},
"fileName": {
"type": "string"
},
"data": {},
"localConfigs": {
"type": "object",
"properties": {
"throttleMs": {
"oneOf": [
{
"type": "number",
"minimum": 0
},
{
"type": "object"
}
]
},
"maxEvents": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "object",
"properties": {
"_gen": {
"type": "string"
}
},
"required": [
"_gen"
]
}
]
},
"discard": {
"type": "object",
"properties": {
"rate": {
"type": "number",
"minimum": 0,
"maximum": 1
}
},
"required": [
"rate"
]
},
"repeat": {
"type": "object",
"properties": {
"rate": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"times": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "object",
"properties": {
"_gen": {
"type": "string"
}
},
"required": [
"_gen"
]
}
]
}
},
"required": [
"rate",
"times"
]
},
"maxHistoryEvents": {
"type": "integer",
"minimum": 0
},
"maxMs": {
"type": "integer",
"minimum": 0
},
"time": {
"type": "integer"
},
"events": {
"type": "object",
"properties": {
"exactly": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "object",
"properties": {
"_gen": {
"type": "string"
}
},
"required": [
"_gen"
]
}
]
}
}
},
"delay": {
"type": "object",
"properties": {
"rate": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"ms": {
"oneOf": [
{
"type": "integer",
"minimum": 0
},
{
"type": "object",
"properties": {
"_gen": {
"type": "string"
}
},
"required": [
"_gen"
]
}
]
}
},
"required": [
"rate",
"ms"
]
},
"history": {
"type": "object",
"properties": {
"events": {
"type": "object",
"properties": {
"max": {
"type": "integer",
"minimum": 0
}
}
}
}
},
"throttle": {
"type": "object",
"properties": {
"ms": {
"oneOf": [
{
"type": "number",
"minimum": 0
},
{
"type": "object"
}
]
}
}
},
"timeMultiplier": {
"oneOf": [
{
"type": "number"
},
{
"type": "object",
"properties": {
"_gen": {
"type": "string"
}
},
"required": [
"_gen"
]
}
]
}
}
},
"fileConfigs": {
"type": "object",
"properties": {
"format": {
"type": "string",
"enum": [
"json",
"jsonl"
]
},
"pretty": {
"type": "boolean"
}
},
"required": [
"format"
]
}
},
"required": [
"fileName",
"data",
"fileConfigs"
]
}