[agustin18]

Summary

Fixes #895 by ensuring carriage returns (\r\n or lone \r) embedded inside quoted CSV/TSV fields are preserved rather than silently rewritten to \n.

Root Cause

When opening a byte stream for CSV/TSV decoding in rows_from_file() (sqlite_utils/utils.py) and the CLI insert command (sqlite_utils/cli.py), io.TextIOWrapper was initialized without an explicit newline argument. By default, io.TextIOWrapper enables universal newline mode (newline=None), translating any \r\n or lone \r into \n as bytes are read.

Because this translation happens underneath csv.reader / csv.DictReader, carriage returns inside quoted fields are destroyed before the CSV parser ever sees them.

Per the Python standard library documentation for the csv module:

"If csvfile is a file object, it should be opened with newline=''. If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly..."

Changes

1. sqlite_utils/utils.py:
   • In rows_from_file() under format == Format.CSV: passed newline="" to io.TextIOWrapper(fp, encoding=use_encoding, newline=""). (Note: Format.TSV delegates to rows_from_file(..., format=Format.CSV) so it is also fixed).
2. sqlite_utils/cli.py:
   • In insert_upsert_implementation(): set newline="" when csv or tsv is true for both sniff-buffered and unbuffered io.TextIOWrapper instances.
3. Tests:
   • In tests/test_rows_from_file.py: Added parameterized regression test test_rows_from_file_preserves_carriage_returns_in_quoted_fields verifying Format.CSV and Format.TSV with both CRLF (\r\n) and lone CR (\r) inside quoted fields.
   • In tests/test_cli_insert.py: Added parameterized test test_insert_csv_tsv_preserves_carriage_returns_in_quoted_fields validating that sqlite-utils insert --csv and --tsv preserve embedded carriage returns when inserting into SQLite tables.

Verification

• Tested with pytest tests/test_rows_from_file.py tests/test_cli_insert.py: all 86 tests passed.
• Full pytest test suite: 1,502 passed, 0 failures.
• Formatting & linting: black --check, flake8, and mypy all clean with 0 errors.

---

📚 Documentation preview 📚: https://sqlite-utils--897.org.readthedocs.build/en/897/

[Muguett-DBY]

The fix also works through format auto-detection and insert --sniff in my checks. I exercised CSV and TSV with both embedded CRLF and lone CR: the base rewrites the cell content in all 16 cases, while this head preserves it. The --text control still normalizes newlines as before.

Checked the exact rows_from_file and CLI implementation functions from 6a7f166 in an existing sqlite-utils 4.2.1 environment, using stdin imports and real SQLite files. Codex-assisted; this wasn't a full branch installation or pytest run.