Excel Data Workflow: Merge, Deduplicate, then Generate SQL

2026-07-17

Why a chained workflow

Real imports rarely start from one clean file:

  1. Marketing, sales, and support each export an Excel file
  2. After combining, the same email or order ID appears twice
  3. Loading without dedup causes PK conflicts or double counts

With ComTools you can run: MergeDedupExcel to SQL, then execute the script on a test database.

Multiple xlsx files
    ↓  Excel Merge (all sheets into one workbook)
Master workbook (pick the sheet you need)
    ↓  Remove Duplicates
Clean sheet
    ↓  Excel to SQL (INSERT)
.sql → test DB → production

Step 1: Merge Excel files

Open Excel Merge:

  1. Select multiple .xlsx / .xlsm (or Add more).
  2. Reorder with ↑↓ (first file comes first).
  3. Start merge and download merged.xlsx.

Note: Merge keeps every worksheet in one workbook; it does not append rows into a single sheet. For same-schema row unions, paste into one sheet in Excel after merge, or INSERT each file separately in the database.

Guide: Excel Merge.

Step 2: Deduplicate by column

Open Remove Duplicates:

  1. Upload the merged file; pick the target sheet.
  2. Check key columns (e.g. Email, OrderId); leave none for whole-row match.
  3. Keep first or last duplicate.
  4. Optional: trim whitespace, ignore case.
  5. Preview → download clean xlsx/csv.
Scenario Tip
Customer emails Key = Email, keep first
Unique order IDs Key = OrderId
Inspect before delete Output mode = mark duplicates
Unique IDs for IN queries Unique values → Text to SQL

Guide: Remove duplicates.

Step 3: Generate INSERT scripts

Open Excel to SQL:

  1. Upload the clean file; select the sheet.
  2. Statement type Insert; set table name (e.g. dbo.Customers).
  3. Choose database dialect.
  4. Map columns; enable batch INSERT when possible.
  5. Copy or download .sql; run on a test database first.

For updates or deletes, use UPDATE / DELETE. Encoding and batching: CSV import tips.

Guide: INSERT from Excel.

Optional middle steps

Step When
CSV to Excel Sources are CSV
Excel Format Polish a shareable intermediate file
Excel Compare Diff before/after dedup
Excel Split Process only one business sheet

Checklist

[ ] Merge order and sheet names look right
[ ] Dedup keys match business keys; preview counts make sense
[ ] INSERT column mapping matches the table
[ ] Test DB row counts match Excel
[ ] Production backup taken; run in batches

FAQ

Dedup fails after merge?
Confirm the sheet has a header row and data.

Must I run all three?
Skip merge for a single file; skip dedup if already unique; use JSON export instead of SQL for frontend mocks.

True row-append across files?
Merge is sheet-based. Same-schema files: dedup each → INSERT each, or append rows in Excel then convert once.


中文版