Skip to main content

Dataset

Dataset is a storage for managing structured tabular data.

The dataset class provides a high-level interface for storing and retrieving structured data with consistent schema, similar to database tables or spreadsheets. It abstracts the underlying storage implementation details, offering a consistent API regardless of where the data is physically stored.

Dataset operates in an append-only mode, allowing new records to be added but not modified or deleted after creation. This makes it particularly suitable for storing crawling results and other data that should be immutable once collected.

The class provides methods for adding data, retrieving data with various filtering options, and exporting data to different formats. You can create a dataset using the open class method, specifying either a name or ID. The underlying storage implementation is determined by the configured storage client.

Usage

from crawlee.storages import Dataset

# Open a dataset
dataset = await Dataset.open(name='my_dataset')

# Add data
await dataset.push_data({'title': 'Example Product', 'price': 99.99})

# Retrieve filtered data
results = await dataset.get_data(limit=10, desc=True)

# Export data
await dataset.export_to('results.json', content_type='json')

Hierarchy

Index

Methods

Properties

Methods

__init__

  • __init__(client, id, name): None

  • Initialize a new instance.

    Preferably use the Dataset.open constructor to create a new instance.

    Parameters

    • client: DatasetClient

      An instance of a storage client.

    • id: str

      The unique identifier of the storage.

    • name: str | None

      The name of the storage, if available.

    Returns None

drop

  • async drop(): None

  • Drop the storage, removing it from the underlying storage client and clearing the cache.

    Returns None

export_to

  • async export_to(key: str, content_type?: Literal[json, csv], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, kwargs: Any): None
  • async export_to(key: str, content_type: Literal[json], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, *: , skipkeys: NotRequired[bool], ensure_ascii: NotRequired[bool], check_circular: NotRequired[bool], allow_nan: NotRequired[bool], cls: NotRequired[type[json.JSONEncoder]], indent: NotRequired[int], separators: NotRequired[tuple[str, str]], default: NotRequired[Callable], sort_keys: NotRequired[bool]): None
  • async export_to(key: str, content_type: Literal[csv], to_kvs_id?: str | None, to_kvs_name?: str | None, to_kvs_storage_client?: StorageClient | None, to_kvs_configuration?: Configuration | None, *: , dialect: NotRequired[str], delimiter: NotRequired[str], doublequote: NotRequired[bool], escapechar: NotRequired[str], lineterminator: NotRequired[str], quotechar: NotRequired[str], quoting: NotRequired[int], skipinitialspace: NotRequired[bool], strict: NotRequired[bool]): None

  • Export the entire dataset into a specified file stored under a key in a key-value store.

    This method consolidates all entries from a specified dataset into one file, which is then saved under a given key in a key-value store. The format of the exported file is determined by the content_type parameter. Either the dataset's ID or name should be specified, and similarly, either the target key-value store's ID or name should be used.

    Parameters

    • key: str

      The key under which to save the data in the key-value store.

    • optionalcontent_type: Literal[json, csv] = 'json'

      The format in which to export the data.

    • optionalto_kvs_id: str | None = None

      ID of the key-value store to save the exported file. Specify only one of ID or name.

    • optionalto_kvs_name: str | None = None

      Name of the key-value store to save the exported file. Specify only one of ID or name.

    • optionalto_kvs_storage_client: StorageClient | None = None

      Storage client to use for the key-value store.

    • optionalto_kvs_configuration: Configuration | None = None

      Configuration for the key-value store.

    • kwargs: Any

      Additional parameters for the export operation, specific to the chosen content type.

    Returns None

get_data

  • async get_data(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): DatasetItemsListPage

  • Retrieve a paginated list of items from a dataset based on various filtering parameters.

    This method provides the flexibility to filter, sort, and modify the appearance of dataset items when listed. Each parameter modifies the result set according to its purpose. The method also supports pagination through 'offset' and 'limit' parameters.

    Parameters

    • optionalkeyword-onlyoffset: int = 0

      Skips the specified number of items at the start.

    • optionalkeyword-onlylimit: int | None = 999_999_999_999

      The maximum number of items to retrieve. Unlimited if None.

    • optionalkeyword-onlyclean: bool = False

      Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

    • optionalkeyword-onlydesc: bool = False

      Set to True to sort results in descending order.

    • optionalkeyword-onlyfields: list[str] | None = None

      Fields to include in each item. Sorts fields as specified if provided.

    • optionalkeyword-onlyomit: list[str] | None = None

      Fields to exclude from each item.

    • optionalkeyword-onlyunwind: list[str] | None = None

      Unwinds items by a specified array field, turning each element into a separate item.

    • optionalkeyword-onlyskip_empty: bool = False

      Excludes empty items from the results if True.

    • optionalkeyword-onlyskip_hidden: bool = False

      Excludes fields starting with '#' if True.

    • optionalkeyword-onlyflatten: list[str] | None = None

      Fields to be flattened in returned items.

    • optionalkeyword-onlyview: str | None = None

      Specifies the dataset view to be used.

    Returns DatasetItemsListPage

get_metadata

iterate_items

  • async iterate_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): AsyncIterator[dict[str, Any]]

  • Iterate over items in the dataset according to specified filters and sorting.

    This method allows for asynchronously iterating through dataset items while applying various filters such as skipping empty items, hiding specific fields, and sorting. It supports pagination via offset and limit parameters, and can modify the appearance of dataset items using fields, omit, unwind, skip_empty, and skip_hidden parameters.

    Parameters

    • optionalkeyword-onlyoffset: int = 0

      Skips the specified number of items at the start.

    • optionalkeyword-onlylimit: int | None = 999_999_999_999

      The maximum number of items to retrieve. Unlimited if None.

    • optionalkeyword-onlyclean: bool = False

      Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

    • optionalkeyword-onlydesc: bool = False

      Set to True to sort results in descending order.

    • optionalkeyword-onlyfields: list[str] | None = None

      Fields to include in each item. Sorts fields as specified if provided.

    • optionalkeyword-onlyomit: list[str] | None = None

      Fields to exclude from each item.

    • optionalkeyword-onlyunwind: list[str] | None = None

      Unwinds items by a specified array field, turning each element into a separate item.

    • optionalkeyword-onlyskip_empty: bool = False

      Excludes empty items from the results if True.

    • optionalkeyword-onlyskip_hidden: bool = False

      Excludes fields starting with '#' if True.

    Returns AsyncIterator[dict[str, Any]]

list_items

  • async list_items(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden): list[dict[str, Any]]

  • Retrieve a list of all items from the dataset according to specified filters and sorting.

    This method collects all dataset items into a list while applying various filters such as skipping empty items, hiding specific fields, and sorting. It supports pagination via offset and limit parameters, and can modify the appearance of dataset items using fields, omit, unwind, skip_empty, and skip_hidden parameters.

    Parameters

    • optionalkeyword-onlyoffset: int = 0

      Skips the specified number of items at the start.

    • optionalkeyword-onlylimit: int | None = 999_999_999_999

      The maximum number of items to retrieve. Unlimited if None.

    • optionalkeyword-onlyclean: bool = False

      Return only non-empty items and excludes hidden fields. Shortcut for skip_hidden and skip_empty.

    • optionalkeyword-onlydesc: bool = False

      Set to True to sort results in descending order.

    • optionalkeyword-onlyfields: list[str] | None = None

      Fields to include in each item. Sorts fields as specified if provided.

    • optionalkeyword-onlyomit: list[str] | None = None

      Fields to exclude from each item.

    • optionalkeyword-onlyunwind: list[str] | None = None

      Unwinds items by a specified array field, turning each element into a separate item.

    • optionalkeyword-onlyskip_empty: bool = False

      Excludes empty items from the results if True.

    • optionalkeyword-onlyskip_hidden: bool = False

      Excludes fields starting with '#' if True.

    Returns list[dict[str, Any]]

open

  • async open(*, id, name, alias, configuration, storage_client): Storage

  • Open a storage, either restore existing or create a new one.

    Parameters

    • optionalkeyword-onlyid: str | None = None

      The storage ID.

    • optionalkeyword-onlyname: str | None = None

      The storage name (global scope, persists across runs). Name can only contain letters "a" through "z", the digits "0" through "9", and the hyphen ("-") but only in the middle of the string (e.g. "my-value-1").

    • optionalkeyword-onlyalias: str | None = None

      The storage alias (run scope, creates unnamed storage).

    • optionalkeyword-onlyconfiguration: Configuration | None = None

      Configuration object used during the storage creation or restoration process.

    • optionalkeyword-onlystorage_client: StorageClient | None = None

      Underlying storage client to use. If not provided, the default global storage client from the service locator will be used.

    Returns Storage

purge

  • async purge(): None

  • Purge the storage, removing all items from the underlying storage client.

    This method does not remove the storage itself, e.g. don't remove the metadata, but clears all items within it.

    Returns None

push_data

  • async push_data(data): None

  • Store an object or an array of objects to the dataset.

    The size of the data is limited by the receiving API and therefore push_data() will only allow objects whose JSON representation is smaller than 9MB. When an array is passed, none of the included objects may be larger than 9MB, but the array itself may be of any size.

    Parameters

    • data: list[dict[str, Any]] | dict[str, Any]

      A JSON serializable data structure to be stored in the dataset. The JSON representation of each item must be smaller than 9MB.

    Returns None

Properties

id

id: str

Get the storage ID.

name

name: str | None

Get the storage name.

Page Options