Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/tech-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Edits are written through as they happen. There is no separate "save" action for
- Writes are coalesced per file for 300ms, then flushed as one transaction. The project's timestamp is the time of the edit, so "last modified" does not depend on when the flush ran.
- Pending writes are also flushed when the page is hidden or unloading. A transaction that waits on a read, or on its requests finishing, does not survive the page unloading. So a flush never reads before it writes, and one from a hidden or unloading page commits explicitly. This is why the storage holds the project's name and writes the whole project record each time, rather than reading the record and updating it.
- A flush at any other time checks, in the same transaction, that the project still exists.
- A failed flush is not retried. The in-memory copy still has the edit, the user sees a toast that stays until dismissed ("Browser storage full" when the quota is exceeded), and the error is logged once per session.
- A failed flush is not retried on a timer, which against a full quota would never end. Its edits stay queued and go with the next flush, whatever causes it: a later edit, the page being hidden, or the project being switched. Meanwhile the user sees a toast that stays until dismissed ("Browser storage full" when the quota is exceeded), is warned before closing the tab, and the error is logged once per session.

MicroPython itself is only needed to build a hex file or flash, not to edit. If it fails to load, projects still open and save, and loading is tried again the next time something needs it.

Expand All @@ -66,7 +66,7 @@ MicroPython itself is only needed to build a hex file or flash, not to edit. If
Tabs tell each other about changes over a `BroadcastChannel`, naming the projects affected:

- Every tab refreshes its list of projects.
- A tab with a changed project open reloads it.
- A tab with a changed project open reloads it, after first saving anything it has outstanding. If that cannot be saved, the tab keeps its own copy rather than lose the edit, and stays out of step until the project is next opened in it or the tab is reloaded.
- A tab with a deleted project open stops writing it, and if the editor is showing goes to the home page with a message.

### Without IndexedDB, and embedded
Expand Down Expand Up @@ -102,7 +102,7 @@ Some choices that are not obvious from the code:
- **The editor opens straight to code.** `/project` in a new tab opens the most recent project rather than asking the user to pick one. The editor has always been quick to get into, and some users bookmark it.
- **No project id in the URL.** The tab's project is kept in session storage instead. This may be revisited if projects are ever stored on a server.
- **Each deployment has its own database.** The database name includes the base path, so deployments that share an origin, such as production and beta, keep separate projects.
- **No warning before closing a tab.** The editor used to warn when a tab was closed with changes made since the last hex file was loaded or saved, because closing the tab lost them. With the database every edit is already stored, so there is nothing to warn about. The warning remains where there is no database.
- **No warning before closing a tab.** The editor used to warn when a tab was closed with changes made since the last hex file was loaded or saved, because closing the tab lost them. With the database every edit is already stored, so there is nothing to warn about unless a save has failed. The warning remains where there is no database.
- **Imports never overwrite silently.** With the database, a hex or an idea makes a new project rather than replacing the open one. Without it there is only one project, so the editor asks before replacing one that has been edited: in micro:bit classroom a student's dropped hex would otherwise replace the teacher's starter program without a word. Adding files that would overwrite existing ones asks first in every mode.
- **A project changed in another tab is reloaded, not merged.** The last write wins.

Expand Down
51 changes: 41 additions & 10 deletions src/e2e/storage-errors.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,20 +85,27 @@ test.describe("storage fallback", () => {
});

/**
* Makes the next IndexedDB put throw, then restores it. The first put in a
* project flush is the project record, so this fails the whole flush.
* Makes the next IndexedDB put throw, then restores it, or with `every`
* makes them all throw. The first put in a project flush is the project
* record, so this fails the whole flush.
*/
const injectWriteError = async (
app: App,
errorName: "QuotaExceededError" | "UnknownError"
errorName: "QuotaExceededError" | "UnknownError",
{ every = false }: { every?: boolean } = {}
) => {
await app.page.evaluate((errorName) => {
const original = IDBObjectStore.prototype.put;
IDBObjectStore.prototype.put = function () {
IDBObjectStore.prototype.put = original;
throw new DOMException(`Simulated ${errorName}`, errorName);
};
}, errorName);
await app.page.evaluate(
({ errorName, every }) => {
const original = IDBObjectStore.prototype.put;
IDBObjectStore.prototype.put = function () {
if (!every) {
IDBObjectStore.prototype.put = original;
}
throw new DOMException(`Simulated ${errorName}`, errorName);
};
},
{ errorName, every }
);
};

/**
Expand Down Expand Up @@ -131,6 +138,30 @@ test.describe("storage write errors", () => {
app.page.getByText("An unexpected error occurred")
).toBeHidden();
});

test("warns before closing while an edit has not saved", async ({ app }) => {
await injectWriteError(app, "UnknownError", { every: true });
await app.typeInEditor("A change!");
await expect(
app.page.getByText("Failed to save your project to browser storage")
).toBeVisible();

await app.closeAndExpectBeforeUnloadPrompt();
});

test("does not warn once a later save has succeeded", async ({ app }) => {
// Only the first save fails. typeInEditor's last keys are typed slowly
// enough to be saved again, successfully, after it.
await injectWriteError(app, "UnknownError");
await app.typeInEditor("A change!");
await expect(
app.page.getByText("Failed to save your project to browser storage")
).toBeVisible();
// Past the storage's flush delay, so the last save has finished.
await app.page.waitForTimeout(600);

await app.closeWithoutBeforeUnloadPrompt();
});
});

/**
Expand Down
73 changes: 71 additions & 2 deletions src/fs/indexeddb-storage.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -172,13 +172,79 @@ describe("IndexedDBFSStorage", () => {
expect((await db.get(projectId))?.name).toEqual("Renamed here");
});

it("has unsaved changes after a failed flush, until the next succeeds", async () => {
expect(storage.hasUnsavedChanges).toEqual(false);
vi.spyOn(db, "save").mockRejectedValueOnce(new Error("Simulated"));
await storage.write("main.py", new Uint8Array([1]));
await storage.flush();
expect(storage.hasUnsavedChanges).toEqual(true);

// Saving anything saves what failed too.
await storage.write("other.py", new Uint8Array([2]));
await storage.flush();
expect(storage.hasUnsavedChanges).toEqual(false);
expect(contents(await db.files(projectId))).toEqual({
"main.py": [1],
"other.py": [2],
});
});

it("does not retry a failed flush until something else flushes", async () => {
const save = vi
.spyOn(db, "save")
.mockRejectedValueOnce(new Error("Simulated"));
await storage.write("main.py", new Uint8Array([1]));
await storage.flush();
await new Promise((resolve) => setTimeout(resolve, 60));
expect(save).toHaveBeenCalledTimes(1);

// As when the project is switched.
await storage.dispose();
expect(save).toHaveBeenCalledTimes(2);
expect(contents(await db.files(projectId))).toEqual({ "main.py": [1] });
});

it("sends the newer content if a file changes again after a failed flush", async () => {
vi.spyOn(db, "save").mockRejectedValueOnce(new Error("Simulated"));
await storage.write("main.py", new Uint8Array([1]));
await storage.flush();
await storage.write("main.py", new Uint8Array([2]));
await storage.remove("old.py");
await storage.flush();
expect(contents(await db.files(projectId))).toEqual({ "main.py": [2] });
expect(storage.hasUnsavedChanges).toEqual(false);
});

it("does not write back a project deleted elsewhere", async () => {
await db.delete(projectId);
await storage.write("main.py", new Uint8Array([1]));
await storage.flush();
expect(errors).toHaveLength(1);
expect(await db.get(projectId)).toBeUndefined();
expect(contents(await db.files(projectId))).toEqual({});

// Nor later: an urgent flush does not check the project exists, so what
// that flush carried must not be kept for it.
window.dispatchEvent(new Event("pagehide"));
await storage.flush();
expect(await db.get(projectId)).toBeUndefined();
expect(contents(await db.files(projectId))).toEqual({});
expect(storage.hasUnsavedChanges).toEqual(false);
});

it("stamps the project with its latest edit, even after failed flushes", async () => {
const now = vi.spyOn(Date, "now");
vi.spyOn(db, "save").mockRejectedValueOnce(new Error("Simulated"));
now.mockReturnValue(100);
await storage.write("main.py", new Uint8Array([1]));
await storage.flush();

now.mockReturnValue(5000);
await storage.write("other.py", new Uint8Array([2]));
await storage.flush();
now.mockRestore();

expect((await db.get(projectId))?.timestamp).toEqual(5000);
});

it("writes nothing once abandoned", async () => {
Expand All @@ -192,7 +258,7 @@ describe("IndexedDBFSStorage", () => {
expect(errors).toEqual([]);
});

it("reports a failed flush and drops those changes rather than throwing", async () => {
it("reports a failed flush rather than throwing, and keeps its changes for the next", async () => {
vi.spyOn(db, "save").mockRejectedValueOnce(
new DOMException("full", "QuotaExceededError")
);
Expand All @@ -203,6 +269,9 @@ describe("IndexedDBFSStorage", () => {

await storage.write("other.py", new Uint8Array([2]));
await storage.flush();
expect(contents(await db.files(projectId))).toEqual({ "other.py": [2] });
expect(contents(await db.files(projectId))).toEqual({
"main.py": [1],
"other.py": [2],
});
});
});
81 changes: 64 additions & 17 deletions src/fs/indexeddb-storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@
*
* SPDX-License-Identifier: MIT
*/
import { FileChanges, ProjectMeta, ProjectsDatabase } from "./projects-db";
import {
FileChanges,
NoSuchProjectError,
ProjectMeta,
ProjectsDatabase,
} from "./projects-db";
import { FSStorage } from "./storage";

const defaultFlushDelayMs = 300;
Expand All @@ -21,9 +26,10 @@ const defaultFlushDelayMs = 300;
* decide, from when the project is opened. Another tab changing it is told
* to this one by Projects, which opens the project afresh.
*
* Failures never propagate: the in-memory primary still holds the content,
* so a failed flush is reported through onError and the changes are dropped
* rather than retried forever against, say, a full quota.
* Failures never propagate: a failed flush is reported through onError. Its
* changes stay queued and go with the next flush, whatever causes that: a
* later edit, the page being hidden, or the project being switched. There
* is no retry of its own, which against a full quota would never end.
*
* The dirty flag is not stored: it exists to warn before work is lost, and
* a project in the database outlives the tab.
Expand All @@ -33,14 +39,19 @@ export class IndexedDBFSStorage implements FSStorage {
private name: string | undefined;
// The name was set here and has not been flushed yet.
private nameUnsaved = false;
// Changes not yet known to be in the database. An entry stays here while
// a flush carrying it is in flight, and after one that fails.
private pendingWrites = new Map<string, Uint8Array | null>();
// When the oldest pending change was made, or undefined with nothing to
// flush: the project's "last modified" is the edit, not the flush, which
// may come later than another project's.
private pendingSince: number | undefined;
// There is something that no flush has yet been sent for, or that one
// failed to save.
private needsFlush = false;
// When the latest change was made. The project's "last modified" is the
// edit, not the flush, which may come later than another project's.
private lastChange = 0;
private timer: ReturnType<typeof setTimeout> | undefined;
private flushing: Promise<void> = Promise.resolve();
private abandoned = false;
private lastFlushFailed = false;
private readonly handleVisibilityChange = () => {
if (document.visibilityState === "hidden") {
void this.flush({ urgent: true });
Expand All @@ -51,7 +62,7 @@ export class IndexedDBFSStorage implements FSStorage {

/**
* @param project The project's record as it is in the database now.
* @param onError Reports a failed flush; the changes in it are dropped.
* @param onError Reports a failed flush; its changes go with the next.
* @param onChange Called after a successful flush, for cross-tab sync.
*/
constructor(
Expand Down Expand Up @@ -92,7 +103,17 @@ export class IndexedDBFSStorage implements FSStorage {
clearTimeout(this.timer);
this.timer = undefined;
this.pendingWrites = new Map();
this.pendingSince = undefined;
this.needsFlush = false;
this.lastFlushFailed = false;
}

/**
* True while the last flush failed, so that closing the tab now would
* lose what it carried. Each flush sends everything still unsaved, so the
* next one to succeed clears this.
*/
get hasUnsavedChanges(): boolean {
return this.lastFlushFailed;
}

/**
Expand Down Expand Up @@ -179,7 +200,8 @@ export class IndexedDBFSStorage implements FSStorage {
this.pendingWrites = new Map();
return;
}
this.pendingSince ??= Date.now();
this.needsFlush = true;
this.lastChange = Date.now();
if (this.timer === undefined) {
this.timer = setTimeout(() => void this.flush(), this.flushDelayMs);
}
Expand All @@ -202,27 +224,52 @@ export class IndexedDBFSStorage implements FSStorage {
flush({ urgent = false }: { urgent?: boolean } = {}): Promise<void> {
clearTimeout(this.timer);
this.timer = undefined;
if (this.pendingSince === undefined) {
if (this.abandoned || !this.needsFlush) {
return this.flushing;
}
const project: ProjectMeta = {
id: this.projectId,
name: this.name,
timestamp: this.pendingSince,
timestamp: this.lastChange,
};
const sent = new Map(this.pendingWrites);
const files: FileChanges = { writes: {}, deletes: [] };
for (const [name, data] of this.pendingWrites) {
for (const [name, data] of sent) {
if (data === null) {
files.deletes.push(name);
} else {
files.writes[name] = data;
}
}
this.pendingWrites = new Map();
this.pendingSince = undefined;
this.nameUnsaved = false;
this.needsFlush = false;
const saved = this.db
.save(project, files, { urgent })
.then(
() => {
for (const [name, data] of sent) {
// Unless it has been changed again since.
if (this.pendingWrites.get(name) === data) {
this.pendingWrites.delete(name);
}
}
if (this.name === project.name) {
this.nameUnsaved = false;
}
this.lastFlushFailed = false;
},
(e) => {
if (e instanceof NoSuchProjectError) {
// Nothing queued must be sent again: an urgent flush does not
// check the project exists and would write part of it back.
this.abandon();
} else if (!this.abandoned) {
// So that the next flush, whatever causes it, sends this again.
this.needsFlush = true;
this.lastFlushFailed = true;
}
throw e;
}
)
.then(() => this.onChange())
.catch((e) => this.onError(e));
const earlier = this.flushing;
Expand Down
30 changes: 28 additions & 2 deletions src/fs/projects-db.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,12 @@
*/
import "fake-indexeddb/auto";
import { openDB } from "idb";
import { databaseName, ProjectMeta, ProjectsDatabase } from "./projects-db";
import {
databaseName,
NoSuchProjectError,
ProjectMeta,
ProjectsDatabase,
} from "./projects-db";

let counter = 0;
const uniqueName = () => `test-${Date.now()}-${counter++}`;
Expand Down Expand Up @@ -94,7 +99,7 @@ describe("ProjectsDatabase", () => {
it("save refuses a project that no longer exists and writes nothing", async () => {
await expect(
db.save(meta("gone", 1), { writes: { "main.py": bytes(1) }, deletes: [] })
).rejects.toThrow(/No such project gone/);
).rejects.toBeInstanceOf(NoSuchProjectError);
expect(await db.get("gone")).toBeUndefined();
expect(contents(await db.files("gone"))).toEqual({});
});
Expand All @@ -120,6 +125,27 @@ describe("ProjectsDatabase", () => {
expect(contents(await db.files("a"))).toEqual({ "main.py": [2] });
});

it("leaves no unhandled rejection when a request in a transaction fails", async () => {
await db.create(meta("a", 1), {});
const unhandled: unknown[] = [];
const listener = (reason: unknown) => unhandled.push(reason);
process.on("unhandledRejection", listener);
// The next put fails as a request, as a put can when storage is full,
// rather than as a call: it adds a key that already exists.
const put = IDBObjectStore.prototype.put;
IDBObjectStore.prototype.put = function (this: IDBObjectStore, value) {
IDBObjectStore.prototype.put = put;
return this.add(value);
};

await expect(db.create(meta("a", 2), {})).rejects.toThrow();
await new Promise((resolve) => setTimeout(resolve, 20));

process.off("unhandledRejection", listener);
expect(unhandled).toEqual([]);
expect(await db.get("a")).toEqual(meta("a", 1));
});

it("touch makes a project the most recent", async () => {
await db.create(meta("a", 1), {});
await db.create(meta("b", 2), {});
Expand Down
Loading