Skip to content

[FEATURE] Keep showing the last successful feed and allow fetching via scheduler - #18

Merged
ErHaWeb merged 5 commits into
ErHaWeb:mainfrom
joh-gp:feature/5-scheduler-import
Nov 8, 2026
Merged

ErHaWeb merged 5 commits into
ErHaWeb:mainfrom
joh-gp:feature/5-scheduler-import

Conversation

@joh-gp

@joh-gp joh-gp commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #5

Temporarily unavailable feeds no longer lead to an error in the frontend. Every successful response is stored per feed URL and survives TYPO3 cache flushes. A failed fetch keeps the stored response.

The setting fetchMode decides per plugin who fetches:

  • live (default): page rendering fetches once cacheDuration has passed. Page-cache lifetimes stay as they were.
  • scheduler: page rendering uses the stored response and feeddisplay:refresh updates it. A missing response is fetched once during rendering. Scheduled fetches read the configured URL directly and are limited to 2 MiB and ten seconds per request.

feeddisplay:remove --url=… removes explicitly selected sources.

Upgrade notes

  • A database schema update is required (new table tx_feeddisplay_feed).
  • Existing plugins stay in live mode, no scheduler job is required.

The original contribution is preserved as tag pr-18-original.

grs and others added 5 commits November 8, 2026 08:09
Use the SimplePie initialization result so failed requests render the
configured error output even when feed getters are selected.

Related: ErHaWeb#5
Store responses as binary documents with their status, selected
headers, MIME type and final URIs. Feeds need not be UTF-8, and
character sets and relative links depend on this context.

Keep serving the last successful response after failed updates, and
keep rendering when the table is missing or a write fails. Share
source freshness between plugin configurations, repair unusable saved
documents and read payloads only on display-cache misses. Page-cache
lifetimes of live plugins stay unchanged.

Co-authored-by: grs <grs@fmo.de>
Use the shared persistent source for scheduled rendering. Bootstrap a
missing response once in the frontend and refresh configured or stored
sources through the schedulable command afterwards.

Limit these fetches in size and time and read only the configured URL,
so a slow or large source cannot hold up the job. Retry failed page
invalidation and discard newly cached pages whose rendered source
revisions have already changed, so a refresh during rendering cannot
leave outdated pages behind.

Resolves: ErHaWeb#5

Co-authored-by: grs <grs@fmo.de>
Remove sources by name instead of by age. Rendered pages can stay
cached far longer than any usage window, so a source in use would look
unused, and stored responses matter most during outages.

Run under the refresh lock and invalidate tagged scheduler pages,
including pages rendered concurrently with removal. Keep the command
out of Scheduler selections on TYPO3 13 and 14.
@ErHaWeb
ErHaWeb force-pushed the feature/5-scheduler-import branch from 2130be4 to 4a76b00 Compare November 8, 2026 22:25
@ErHaWeb
ErHaWeb merged commit b31c7a7 into ErHaWeb:main Nov 8, 2026
15 checks passed
@ErHaWeb

ErHaWeb commented Nov 8, 2026

Copy link
Copy Markdown
Owner

@joh-gp Thanks a lot for this! While testing I found a few things I wanted to fix before releasing it, so I reworked it on your branch. Feeds that aren't UTF-8 are now stored correctly, relative links still work after a redirect, and existing live pages keep their cache lifetime. I also replaced feeddisplay:prune with an explicit feeddisplay:remove, because a page that stays cached for a long time could make a feed look unused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import feeds through scheduler task

2 participants