Sample architecture

metoda has built several products on Price API. Here is how we integrated the API.

Database of jobs and their state

When we create a job with Price API, we store the job_id together with all the parameters (like source, country, topic, key, ...) in a database table.

This table also keeps track of the status as it was last reported and whether it was processed and when to next try processing it.

Scheduler

We have a process that identifies data that needs to be updated. It then creates a job with Price API and keeps track of it in aforementioned database table.

This process also makes sure not to request data a second time when it was already requested and the job has not finished yet.

We implement this either as a long-running loop or with something like CRON.

Processor

A separate process regularly queries the database for jobs that are ready to be acted on:

• For jobs that have a status neither finished nor cancelled, request a status update after a minute.
• For jobs that have a status finished, try to download and process the results.
• For jobs that have a status cancelled take appropriate action, like retrying or alerting.

We implement this either as a long-running loop or with something like CRON.