Skip to main content

SmartApifyStorageClient

Storage client that automatically selects cloud or local storage client based on the environment.

This storage client provides access to datasets, key-value stores, and request queues by intelligently delegating to either the cloud or local storage client based on the execution environment and configuration.

When running on the Apify platform (which is detected via environment variables), this client automatically uses the cloud_storage_client to store storage data there. When running locally, it uses the local_storage_client to store storage data there. You can also force cloud storage usage from your local machine by using the force_cloud argument.

This storage client is designed to work specifically in Actor context and provides a seamless development experience where the same code works both locally and on the Apify platform without any changes.

Index

Methods

Methods

__init__

  • __init__(*, cloud_storage_client, local_storage_client): None

  • Initialize a new instance.

    Parameters

    • optionalkeyword-onlycloud_storage_client: StorageClient | None = None

      Storage client used when an Actor is running on the Apify platform, or when explicitly enabled via the force_cloud argument. Defaults to ApifyStorageClient.

    • optionalkeyword-onlylocal_storage_client: StorageClient | None = None

      Storage client used when an Actor is not running on the Apify platform and when force_cloud flag is not set. Defaults to FileSystemStorageClient.

    Returns None

__str__

  • __str__(): str

  • Returns str

create_dataset_client

  • async create_dataset_client(*, id, name, alias, configuration): DatasetClient

  • Parameters

    • optionalkeyword-onlyid: str | None = None
    • optionalkeyword-onlyname: str | None = None
    • optionalkeyword-onlyalias: str | None = None
    • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

    Returns DatasetClient

create_kvs_client

  • async create_kvs_client(*, id, name, alias, configuration): KeyValueStoreClient

  • Parameters

    • optionalkeyword-onlyid: str | None = None
    • optionalkeyword-onlyname: str | None = None
    • optionalkeyword-onlyalias: str | None = None
    • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

    Returns KeyValueStoreClient

create_rq_client

  • async create_rq_client(*, id, name, alias, configuration): RequestQueueClient

  • Parameters

    • optionalkeyword-onlyid: str | None = None
    • optionalkeyword-onlyname: str | None = None
    • optionalkeyword-onlyalias: str | None = None
    • optionalkeyword-onlyconfiguration: CrawleeConfiguration | None = None

    Returns RequestQueueClient

get_storage_client_cache_key

  • get_storage_client_cache_key(configuration): Hashable

  • Parameters

    • configuration: CrawleeConfiguration

    Returns Hashable

get_suitable_storage_client

  • get_suitable_storage_client(*, force_cloud): StorageClient

  • Get a suitable storage client based on the global configuration and the value of the force_cloud flag.

    Parameters

    • optionalkeyword-onlyforce_cloud: bool = False

      If True, return cloud_storage_client.

    Returns StorageClient

Page Options