Save Data To Storage

The Storage is used to collect the different outcome data of document processing.

In general case (when /api/process endpoint is used) only the request and response are saved.
When the Complete Server-Side Verification is used (via the /api/v2/transaction endpoint), the Storage keeps the request and response and also the metadata, session keys, and challenges for RFID processing.

Supported storage types:

Amazon S3 or any other S3-compatible storage (for example, MinIO, Ceph, Backblaze, and others)
Google Cloud Storage
Microsoft Azure
File System

For access management, we support Workload Identity Federation; see links with configuration instructions in the table below.

Storage	Configuration instructions
Azure Blob Storage	Azure AD Workload Identity Documentation Support Azure Workload Identities
Google Cloud Storage	Authenticate to Google Cloud APIs from GKE workloads
AWS S3	EKS Pod Identities IAM roles for service accounts

To set up the storage, use:

Amazon S3Google Cloud StorageMicrosoft AzureFile System

config.yaml

service:
  storage:
    type: s3

    s3:
      accessKey: "AKIAIOSFODNN7REGULAEXAMPLE"
      accessSecret: "WJalrXUtnFEMI/K7MDENG/bPxRfiCYREGULAEXAMPLEKEY"
      region: "eu-central-1"
      secure: true
      endpointUrl: "https://s3.eu-central-1.amazonaws.com"

Parameter	Type	Default	Description
`type`	string	—	Set the `s3` value to use Amazon S3
`accessKey`	string	—	Access key for the S3 storage.
`accessSecret`	string	—	Access secret for the S3 storage.
`region`	string	—	Region for the S3 storage.
`secure`	boolean	`true`	Whether to check the CA certificate.
`endpointUrl`	string	—	URL of the local HTTP REST endpoint. For more details see the Amazon documentation about Website endpoints.

Other Amazon S3 Storage authentication methods:

Via the standard AWS environment variables.
By running the application within the AWS environment and assigning the necessary IAM role to your application with the required permissions to access S3.

Note that MinIO storage is utilized for on-premises installations within our containers. It is compatible with AWS S3 and is configured in the same way.

Amazon S3 requires the following permissions:

s3:PutObject
s3:PutObjectAcl
s3:GetObject
s3:GetObjectAcl
s3:DeleteObject
s3:ListObjects

Here is an S3 policy example:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::regula-test-permissions-bucket"
                ]
            },
            {
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:GetObject",
                    "s3:GetObjectAcl",
                    "s3:DeleteObject"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::regula-test-permissions-bucket/*"
                ]
            }
        ]
    }

config.yaml

service:
  storage:
    type: gcs

    gcs:
      gcsKeyJson: "/etc/credentials/gcs_key.json"

Parameter	Type	Default	Description
`type`	string	—	Set the `gcs` value to use Google Cloud Storage
`gcsKeyJson`	string	`"/etc/credentials/gcs_key.json"`	Path to the credentials file for Google Cloud Storage access.

To enable Google Cloud Storage, follow the official Google documentation.

For the required Google Cloud Platform permissions, see the table below.

Bucket permission name	Description
`storage.buckets.get`	Read bucket metadata, excluding IAM policies, and list or read the Pub/Sub notification configurations on a bucket.
`storage.objects.create`	Add new objects to a bucket.
`storage.objects.delete`	Delete objects.
`storage.objects.get`	Read object data and metadata, excluding ACLs.
`storage.objects.list`	List objects in a bucket. Also read object metadata, excluding ACLs, when listing.
`storage.objects.update`	Update object metadata, excluding ACLs. Also read object metadata, excluding ACLs, when updating.

For details see the IAM permissions for Cloud Storage article.

config.yaml

service:
  storage:
    type: az

    az:
      storageAccount: "apidevaccount"
      connectionString: "DefaultEndpointsProtocol=https;AccountName=apidevaccount;AccountKey=************;EndpointSuffix=core.windows.net"

Parameter	Type	Default	Description
`type`	string	—	Set the `az` value to use Microsoft Azure
`storageAccount`	string	`<AZURE_STORAGE_ACCOUNT>`	Name of the Azure Storage account.
`connectionString`	string	`<AZURE_CONNECTION_STRING>`	Connection string for Azure Storage. Due to security reasons, has different order of precedence.

Other Microsoft Azure Storage authentication methods:

Via the standard Azure environment variables. Refer to the official Azure documentation for instructions.
When running in an Azure environment and assigned the appropriate role with Azure Storage access rights, authentication is enabled.

Note that authentication both methods described above require the storage account to be specified in the storage.type.az.storageAccount field.

config.yaml

service:
  storage:
    type: fs

    fs:

Parameter	Type	Default	Env Variable	Description
`type`	string	—	—	Set the `fs` value to use file system
`fs`	string	—	—	To set up using the file system as the storage, leave this section empty and define the parameter `processing` → `results` → `location` → `folder`

Note that if you use Network file storage, you need to mount them as local drives.

Example

Below, see the contents' example of the config.yaml file or download it.

config.yaml

service:
  webServer:
    port: 8080
    workers: 1
    timeout: 30

    demoApp:
      enabled: true

    cors:
      origins: "*"
      headers: "Content-Type"
      methods: "POST,PUT,GET,DELETE,PATCH,HEAD"

    ssl:
      enabled: false
      cert: certs/tls.crt
      key: certs/tls.key
      tlsVersion: 1.2

    logging:
      level: INFO
      formatter: text

      access:
        console: true
        path: logs/docreader-access.log

      app:
        console: true
        path: logs/docreader-app.log

  processing:
    enabled: true
    results:
      location:
        bucket: docreader-processing
        container: docreader-processing
        folder: docreader-processing
        prefix: ""

  storage:
    type: s3

    s3:
      accessKey: minioadmin
      accessSecret: minioadmin
      endpointUrl: http://s3-storage:9000
      region: eu-central-1
      secure: true

If you use Docker Compose, you can mount the config.yaml in the docker-compose.yml file. See the example below or download it.

docker-compose.yml

version: "3.7"

services:
  docreader:
    container_name: docreader
    image: regulaforensics/docreader:latest
    volumes:
      - ./regula.license:/app/extBin/unix/regula.license
      - ./config.yaml:/app/config.yaml
    networks:
      - docreader-network
    healthcheck:
      test: curl -f http://127.0.0.1:8080/api/ping
      interval: 60s
      start_period: 60s
      timeout: 30s
      retries: 5
    depends_on:
      - s3-storage
    ports:
      - "8080:8080"

  # Storage
  s3-storage:
    image: quay.io/minio/minio:RELEASE.2023-10-25T06-33-25Z
    command: server --console-address ":9001" /data
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - minio-data:/data
    networks:
      - docreader-network
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

volumes:
  minio-data:

networks:
  docreader-network:
    driver: bridge

To set the license, place the regula.license file into the same folder where the downloaded docker-compose.yml is located.

To start the Docker container, invoke:

sudo docker-compose up -d

To check the status, run:

sudo docker-compose ps

Next Steps

Web Service Open API Specification