Automating Data Extraction with Schedulers in ShoppingScraper API

Utilize schedulers for automatic, recurring data extraction based on predefined schedules. A scheduler triggers the creation of new batches at each run.

Creating Schedulers

Create schedulers to automate batch operations. Schedulers can be based on EAN codes or a CSV URL. Frequencies are defined using cron schedule expressions.

POST https://api.shoppingscraper.com/schedulers?api_key=your_api_key_here&enforce_validation=false&run_now=true
Payload:
{
  "sites": ["shopping.google.nl"],
  "queries_file": "https://yourcsvfile.com/file.csv",
  "frequencies": ["0 0 28 2 *"],
  "eans": [],
  "name": "Scheduler Name"
}

Response Upon Scheduler Creation

The API returns a payload indicating the creation of both the scheduler and an immediate batch if run_now is set to true.

{
    "scheduler_id": "unique_scheduler_id_here",
    "batch_id": "generated_batch_id_here",
    "total_query_no": 80,
    "estimated_processing_time": 40.0,
    "malformed_queries": []
}

Fetching All Schedulers

Retrieve a list of all schedulers created under your API key.

GET https://api.shoppingscraper.com/schedulers/?api_key=your_api_key_here

Getting Latest Scheduler Results

Download the most recent results of a specific scheduler. The URL for results remains constant; data is updated behind the scenes.

GET https://api.shoppingscraper.com/schedulers/scheduler_id_here/results?api_key=your_api_key_here&output_format=csv

Updating Schedulers

Schedulers can be updated to run immediately or adjust their configurations.

PUT https://api.shoppingscraper.com/schedulers/scheduler_id_here?api_key=your_api_key_here&run_now=true

Deleting Schedulers

Remove a scheduler if it is no longer needed.

DELETE https://api.shoppingscraper.com/schedulers/scheduler_id_here?api_key=your_api_key_here

Note: Replace your_api_key_here, unique_scheduler_id_here, and generated_batch_id_here with your actual API key, scheduler ID, and batch ID respectively.