Filter Dataset with CSV/JSON Files

Create a dataset snapshot based on a provided filter

Paste your API token to the authorization field. To get a token, Create an account and learn how to generate a token.

General Description

• A call to this endpoint starts the async job of filtering the dataset and creating a snapshot with filtered data in your account.
• The maximum amount of time for the job to finish is 5 minutes. If the job doesn’t finish in this timeframe it will be cancelled.
• Creating the dataset snapshot is subject to charges based on the snapshot size and record price.
• The maximum depth of nesting the filter groups is 3.
• You can now upload CSV or JSON files containing filter values for more efficient filtering of large value sets.

Request

Endpoint

POST http://api.brightdata.com/datasets/filter

Query Parameters

Parameter	Type	Description
dataset_id	string	The unique identifier of the dataset to filter
records_limit	integer	Optional. Maximum number of records to include in the result

Headers

Header	Type	Description
Authorization	string	Your API token for authentication
Content-Type	string	Must be set to multipart/form-data when uploading files

Request Body

When using file uploads, the request body should be formatted as multipart/form-data with the following fields:

Field	Type	Description
filter	string	A stringified JSON object containing filter criteria. Reference file names in the value field.
files[]	file	One or more CSV or JSON files containing filter values. Multiple files can be uploaded.

File Format Requirements

CSV File Format

• The first line must be a header matching the field name in your filter
• Each subsequent line contains a single value
• Example (industries.csv):

  industries:value
  Accounting
  Ad Network
  Advertising
  

JSON File Format

• An array of objects where each object has a key matching the field name in your filter
• Example (industries.json):

  [
    {"industries:value": "Accounting"},
    {"industries:value": "Ad Network"},
    {"industries:value": "Advertising"}
  ]
  

Filter Syntax with File References

When using file uploads, reference the filename in your filter’s value field:

{
  "operator": "and",
  "filters": [
    {
      "name": "industries:value",
      "operator": "includes",
      "value": "industries.csv"
    }
  ]
}

Supported Operators for File References

Operator	Field Types	Description
in	Any	Tests if field value equals any value in the file
not_in	Any	Tests if field value doesn’t equal any value in the file
includes	Array, Text	Tests if field value contains any value in the file
not_includes	Array, Text	Tests if field value doesn’t contain any value in the file
array_includes	Array	Tests if any value in the file exists in field value (exact match)
not_array_includes	Array	Tests if no value in the file exists in field value (exact match)

Example Use Case

Filtering Companies by Industry Using a CSV File

CSV File Content (industries.csv)

industries:value
Accounting
Ad Network
Advertising

cURL Request

curl \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: multipart/form-data" \
  -F "files[]=@/path/to/industries.csv" \
  -F "filter={\"operator\":\"and\",\"filters\":[{\"name\":\"industries:value\",\"operator\":\"includes\",\"value\":\"industries.csv\"}]}" \
  "api.brightdata.com/datasets/filter?dataset_id=gd_l1vijqt9jfj7olije"

Using Multiple Files in a Filter

You can upload and reference multiple files in a single request:

cURL Request with Multiple Files

curl \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: multipart/form-data" \
  -F "files[]=@/path/to/industries.csv" \
  -F "files[]=@/path/to/regions.csv" \
  -F "filter={\"operator\":\"and\",\"filters\":[{\"name\":\"industries:value\",\"operator\":\"includes\",\"value\":\"industries.csv\"},{\"name\":\"region\",\"operator\":\"in\",\"value\":\"regions.csv\"}]}" \
  "api.brightdata.com/datasets/filter?dataset_id=gd_l1vijqt9jfj7olije"

Troubleshooting & FAQs

Issue: “File not found” error

Solution: Ensure the filename referenced in your filter exactly matches the uploaded file name.

Issue: “Invalid file format” error

Solution: Check that your CSV has the correct header matching the field name, or your JSON is properly formatted as an array of objects.

Issue: “Field not found” error

Solution: Verify that the field name in your filter exists in the dataset. Use the Get Dataset Metadata endpoint to see available fields.

Related Documentation

• Get Dataset List
• Get Dataset Metadata
• Dataset Download API

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json

dataset_id

string

required

ID of the dataset to filter

filter

object

required

• Single field filter
• Filters group
• Single field filter w/out value

Show child attributes

filter.name

string

required

Field name to filter by

filter.operator

enum<string>

required

Available options:

=,

!=,

>,

<,

>=,

<=,

in,

not_in,

includes,

not_includes,

array_includes,

not_array_includes

filter.value

required

Value to filter by

Example:

{
  "name": "name",
  "operator": "=",
  "value": "John"
}

records_limit

integer

Response

200

application/json

Job of creating the snapshot successfully started

snapshot_id

string

ID of the snapshot