All posts

Extract Paginated Web Tables to CSV in Broma

Extract Paginated Web Tables to CSV in Broma

Paginated tables are one of the most common browser chores: a supplier catalog, a stock list, a public directory, a dashboard report. The task looks simple until page two. Rows duplicate, the Next button disappears, a column shifts because a product name contains a comma, or the flow clicks Next before the table has refreshed.

This guide builds one focused Broma workflow: read a public paginated table, collect rows across pages, and export a clean CSV. It assumes the table is ordinary HTML and the site allows this kind of collection at a polite pace. Check the site terms first, keep the volume reasonable, and do not collect personal data.

Start with the output contract

Before touching selectors, decide what one row means. For a supplier catalog, the contract might be:

  • One row per product variant
  • Columns: SKU, Product, Price, Stock
  • Header row included once
  • CSV fields quoted so commas and line breaks do not shift columns

Write that contract down. It prevents the common failure where the workflow collects visible text but not the fields you actually need.

Pick selectors that describe the table, not its position

Open the page in Broma, use the browser inspector, and look for stable attributes. Good anchors are usually:

#catalog
#catalog tbody tr
td[data-field='sku']
button[aria-label='Next page']
[data-testid='page-number']

Avoid selectors like body > div:nth-child(4) > table. They break when a banner, cookie notice, or layout wrapper appears. If the table has no stable IDs, anchor to the nearest stable container and document the column order in the flow. A selector that is slightly less elegant but easy to repair is better than a clever selector nobody can maintain.

For the Next button, prefer a selector that also exposes state:

button[aria-label='Next page']:not([disabled])

If the site hides the button on the last page, the wait step will fail cleanly instead of clicking a dead control.

Read one page into rows

Add an Evaluate expression step after the page has loaded. This expression returns an array of arrays, one inner array per table row:

(() => {
  const table = document.querySelector('#catalog');
  if (!table) return [];
  return [...table.querySelectorAll('tbody tr')]
    .filter(tr => tr.querySelectorAll('td').length > 1)
    .map(tr => [...tr.querySelectorAll('td')].map(td => td.textContent.trim()));
})()

Why read the whole row at once? It keeps the row cells aligned. If you read each column with a separate step, a missing cell on one row can silently shift every later value. The filter removes spacer or message rows that are not real records.

If the table uses links, textContent still gives you the visible label. If you need the URL, read the href attribute in a separate expression and append it as another column.

Convert rows to CSV safely

Do not join cells with commas directly. Product names often contain commas, quotes, or line breaks. Use a small expression that quotes every field and doubles internal quotes:

const q = String.fromCharCode(34);
const nl = String.fromCharCode(10);
const esc = v => {
  const s = String(v).trim();
  return q + s.split(q).join(q + q) + q;
};
return rows.map(r => r.map(esc).join(',')).join(nl);

In the flow, keep a text variable called csv. Initialize it with the header once:

SKU,Product,Price,Stock

After each page, append a newline and the converted rows. Quoting every data field is acceptable and makes the file predictable. If you need a page number for debugging, add it as a fifth column before conversion.

Loop through pages without fixed sleeps

A fixed wait is the main reason these flows fail. Page two may load in 300 ms or three seconds. Instead of sleeping, wait for evidence that the table changed.

Use this loop shape:

  1. Set page = 1, csv = header, and previousFirstRow = ''.
  2. Read the current table rows and append them to csv.
  3. Save the first row text as previousFirstRow.
  4. Check whether the Next button exists and is enabled. If not, stop.
  5. Click Next.
  6. Wait until the first row is different from previousFirstRow, or until the page indicator changes.
  7. Increment page and repeat, with a maximum page guard such as 50.

The wait condition can be a short expression:

(() => {
  const first = document.querySelector('#catalog tbody tr td');
  if (!first) return false;
  return first.textContent.trim() !== previousFirstRow;
})()

In Broma, compare against the saved variable in the Wait for condition step. If the site exposes a page number, use that instead; it is usually more reliable than row text. Add a one- to two-second pause between pages only if the site expects a polite pace. That pause is not a substitute for the change check.

Export and verify the CSV

Save the csv variable to a file with a Save file step. Then verify three things before trusting the run:

  • The first data row matches the first row you see manually.
  • The last row belongs to the last page, not the second-to-last.
  • The row count is close to pages x rows per page, allowing for the final partial page.

If the count is too high, you probably appended the same page twice. If it is too low, the loop stopped early or the table selector missed rows during a re-render.

Troubleshooting the common failures

Duplicate rows

The click happened before the table refreshed. Wait for the page indicator or first-row change, not for the click itself.

Missing final page

Some sites disable Next by removing the element, others by adding disabled. Check both states. Also confirm that the final page is not loaded by a separate Show more control.

CSV columns shifted

A field contains a comma or newline. Use the quoting expression above and open the file in a spreadsheet to inspect the first few rows.

Selector worked once, then failed

The table re-rendered and replaced the node. Anchor to the stable table container and wait for rows to exist before reading them.

Infinite scroll instead of pagination

This recipe is for explicit pagination. Infinite scroll needs a different pattern: scroll, wait for the row count to increase, and stop when the count stops changing. Do not mix the two approaches.

Where AI helps, and where it does not

You can describe the table and pagination to Broma's AI agent and ask for a selector map and loop plan. Treat that as a draft. Open the page, verify each selector in the inspector, and test the wait condition manually. AI is optional and runs on your own AI account; it is not included in the Broma price.

Once the flow works, keep the selectors in one small variable block at the top. When the site changes, one edit fixes the whole workflow. That is the difference between a one-off extraction and a reusable operations tool.