S3 Proxy

Implementation

The server is implemented using the HTTP2 Framework.

A local cache of the underlying S3 files is maintained to avoid repeated fetches from S3. The local storage location can be controlled via the --cache-dir option when starting the server. A TTL value is set on the cached files. The duration can be controlled via the --ttl option.

Configuration

The server can be configured through the following command line options:

• --auth-key - A bearer token value to use to access internal management API endpoints.

  • At present the only management endpoint is /_proxy/_private/_cache/clear to clear the cache.

• --key - The AWS Access key ID to use to authorise the AWS SDK.

• --secret - The AWS Secret access key to use to authorise the AWS SDK.

• --region - The AWS Region in which the bucket exists.

• --bucket - The target AWS S3 Bucket from which objects are retrieved.

• --port - The port on which the server listens. Default 8000.

• --threads - Number of worker threads for the server. Defaults to std::thread::hardware_concurrency.

• --ttl - Cache expiry TTL in seconds. Default 300. All S3 objects are cached for efficiency.

• --cache-dir - The directory under which cached S3 objects are stored.

  • Objects are stored under a filename that is the result of invoking std::hash<std::string> on the full path of the object.

  • Since hash values are used as filenames, the directory structure will be flat. Makes it easy to have external jobs (eg. cron jobs) that delete the files on a regular basis. No real need for it, unless a lot of files have been removed from the backing S3 bucket.

• --reject-query-strings - A flag to indicate that requests that contain query strings should be rejected as bad request. Query strings in the context of serving out S3 resources may be viewed as a malicious request.

• --console - Set to true to echo log messages to stdout as well. Default false.

• --log-level - Set the desired log level for the service. Valid values are one of critical, warn, info or debug.

• --dir - Directory under which log files are to be stored. Log files will be rotated daily. Default logs/ directory under the current working directory.

  • Note: if the directory does not exist, no logs will be output.

  • A trailing / is mandatory for the specified directory (eg. /var/logs/proxy/).

  • External scheduled job (cron or similar) that keeps a bound on the number of log files is required.

    • If run as a Docker container without the log directory being mounted as a volume, restarting the container will clear logs. This will delete all historical logs.

    • Since we are dealing with a backing S3 bucket, the proxy will almost certainly be run on AWS infrastructure. In this case, it is safe to restart the container, since the logs should end up in CloudWatch.

      • Ensure the Docker daemon has been configured to log to CloudWatch.

      • Ensure the server is started with --console true.

• --mmdb-host - If integration with mmdb-ws is desired, specify the hostname for the service.

• --mmdb-port - Port on which the mmdb-ws service is listening. Default is 2010. Only relevant if --mmdb-host is specified.

• --ilp-host - If metrics are to be published to an ILP (Influx Line Protocol) compatible database such as QuestDB. Works in combination with --mmdb-host. Disabled if not specified.

• --ilp-port - Port on which ILP TCP service listens. Default is 9009 (as used by QuestDB).

• --ilp-series-name - Name for the time series for metrics. Default is request.

• --mongo-uri - If metrics are to be published to MongoDB. Should follow Connection String URI format. Works in combination with --mmdb-host. Disabled if not specified.

• --mongo-database - Mongo database to write metrics to. Default is metrics.

• --mongo-collection - Mongo collection to write metrics to. Default is request.

Full list of options can always be consulted by running the following:

Metric

Metric is captured in a simple struct. If MMDB integration is enabled, additional information about the visitors geo-location is retrieved and added to the metric. The combined result is then saved to Time series database over ILP and/or MongoDB.