Skip to content

feat(core): exclude posts tagged #no-search-index from the index#52

Merged
betschki merged 2 commits into
mainfrom
feat/exclude-tags-no-search-index
Jun 26, 2026
Merged

feat(core): exclude posts tagged #no-search-index from the index#52
betschki merged 2 commits into
mainfrom
feat/exclude-tags-no-search-index

Conversation

@betschki

@betschki betschki commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

What

Adds a configurable, tag-based way to keep individual posts out of the search index — for landing pages, legal/policy pages, internal notes, etc. By default, any post tagged #no-search-index is excluded.

This is the same category of index-time content policy that already lives in packages/coreisInternalTag() (exclude internal tags) and buildRedactedPost() (redact gated bodies) — so it sits right next to them, as one canonical, documented convention rather than something each consumer reinvents.

How

  • collection.excludeTags (config) — optional string[]; the indexer defaults to ['#no-search-index'] when omitted, and an explicit [] disables exclusion.
  • isExcludedByTag() (core) — mirrors isInternalTag's defensive shape: matches a post's tags by name or slug, case-insensitively, and derives the Ghost hash- slug from #-prefixed names (so #no-search-index also matches hash-no-search-index even if the tag's display name was changed).
  • indexAllPosts filters excluded posts out of the bulk sync.
  • indexPost deletes-and-returns when the tag is present — so through the webhook handler (which calls indexPost), an edit that adds the tag de-indexes the post. No webhook-handler change needed.
  • The default lives in core so every consumer (CLI, webhook handler) shares the convention without opting in.

Tests

  • config: excludeTags undefined when omitted, accepts a custom list, accepts [], and createDefaultConfig surfaces the convention.
  • core: unit tests for isExcludedByTag (default match by name/slug, custom list override, [] disables), plus indexPost integration proving a #no-search-index post is deleted, not upserted (and indexed normally when excludeTags: []).
  • Full monorepo green: config 18 · core 20 · search-ui 52 · webhook 5 · cli 3; typecheck 7/7; lint 0 errors.

Notes

  • No version bump here — feature only; bump/release separately when shipping.
  • Convergence (out of scope for this PR): the package isn't the only indexer. The portal (apps/api/src/lib/typesense/*) and an ad-hoc thepulse script each reimplement indexing and have already drifted (the internal-tag leak came from one such fork). Mirroring this exclusion there — ideally converging thepulse onto this package's CLI — is the bigger win and a separate task.

Summary by Sourcery

Introduce configurable tag-based exclusion of posts from the search index, defaulting to ignoring posts tagged #no-search-index across all core indexer entry points.

New Features:

  • Allow collection configs to specify excludeTags so posts with certain tags are omitted from the Typesense search index.
  • Default to excluding posts tagged #no-search-index, applied consistently in bulk indexing and single-post indexing paths.

Enhancements:

  • Ensure indexPost removes previously indexed documents when a post becomes excluded by tag instead of upserting them.
  • Share the #no-search-index exclusion convention via createDefaultConfig so all consumers inherit the same content policy.

Documentation:

  • Document tag-based exclusion, including the #no-search-index default and excludeTags configuration, in the README.
  • Clarify how tag names and slugs are matched case-insensitively for exclusion.

Tests:

  • Add unit and integration tests for tag-based exclusion behaviour, including default #no-search-index handling, custom excludeTags lists, and exclusion disabling with an empty list.
  • Extend config tests to cover validation and defaulting for the excludeTags collection option.

Summary by CodeRabbit

  • New Features

    • Added support for excluding individual posts from search indexing using tags, with a default #no-search-index convention.
    • Posts with matching tags are now automatically removed from search if they become excluded later.
    • You can now override or disable this behavior with a collection-level exclusion list, including case-insensitive tag matching.
  • Documentation

    • Expanded the README with guidance on excluding specific posts and how hidden #-prefixed tags behave.

Add a configurable, tag-based content-policy exclusion
(`collection.excludeTags`, default `['#no-search-index']`) alongside the
existing internal-tag filtering and gated-content redaction. Posts carrying an
excluded tag are skipped by indexAllPosts and removed by indexPost — so via the
webhook handler, an edit that adds the tag de-indexes the post.

Matching mirrors isInternalTag: by tag name or slug, case-insensitively,
deriving the Ghost `hash-` slug from `#`-prefixed names. The default lives in
core so every consumer (CLI, webhook handler) shares one canonical convention;
an explicit `[]` disables it. Documented in the README + createDefaultConfig.
@sourcery-ai

sourcery-ai Bot commented Jun 26, 2026

Copy link
Copy Markdown

Reviewer's Guide

Adds a configurable tag-based mechanism to exclude posts (by default those tagged #no-search-index) from the Typesense search index at both bulk and single-post indexing paths, wires the configuration through the shared config package, and documents the behavior in the README with accompanying unit tests.

Sequence diagram for indexPost handling of #no-search-index exclusion

sequenceDiagram
    actor Ghost
    participant WebhookHandler
    participant GhostTypesenseManager
    participant TypesenseCollection

    Ghost ->> WebhookHandler: post.updated
    WebhookHandler ->> GhostTypesenseManager: indexPost(post)
    GhostTypesenseManager ->> GhostTypesenseManager: isExcludedByTag(post.data)
    alt [post is excluded by tag]
        GhostTypesenseManager ->> TypesenseCollection: documents().delete(filter_by: id:postId)
        TypesenseCollection -->> GhostTypesenseManager: delete result
        GhostTypesenseManager -->> WebhookHandler: return
    else [post is not excluded]
        GhostTypesenseManager ->> TypesenseCollection: documents().upsert(document)
        TypesenseCollection -->> GhostTypesenseManager: upsert result
        GhostTypesenseManager -->> WebhookHandler: return
    end
Loading

File-Level Changes

Change Details Files
Introduce tag-based exclusion logic in the core indexer and apply it to both bulk and single-post indexing operations.
  • Add DEFAULT_EXCLUDE_TAGS constant defaulting to ['#no-search-index'] in the core indexer.
  • Implement private isExcludedByTag method that derives the active exclude tag list from collection.excludeTags (or the default), normalizes tags case-insensitively, and matches against both tag names and slugs including Ghost-style hash- slugs for #‑prefixed tags.
  • Update indexAllPosts to filter all fetched posts through isExcludedByTag before applying the existing gated-content visibility filter, and log counts based on the filtered set.
  • Update indexPost to early-return by deleting an existing document (and skipping upsert) when isExcludedByTag marks the post as excluded, ensuring webhook-driven edits that add the tag de-index the post.
packages/core/src/index.ts
Extend the shared config schema to support excludeTags and propagate the #no-search-index convention into default configs.
  • Add excludeTags?: string[] to CollectionConfigSchema with documentation describing its semantics, keeping it optional so existing configs remain valid and clarifying that [] disables exclusion.
  • Update createDefaultConfig to include excludeTags: ['#no-search-index'] in the generated collection configuration as the canonical convention.
packages/config/src/index.ts
Add tests covering excludeTags configuration behavior and core tag-based exclusion logic, including webhook semantics.
  • Extend the mocked Ghost API read() path in core tests to return an excluded-post-1 fixture carrying the #no-search-index/hash-no-search-index tag pair.
  • Add a new GhostTypesenseManager — tag-based exclusion (#no-search-index) test suite exercising isExcludedByTag via a private helper, covering default matching by name/slug, custom excludeTags list overriding the default, and empty excludeTags disabling exclusion.
  • Add indexPost integration tests verifying that an excluded post is deleted (not upserted) and that the same post is indexed normally when excludeTags is set to [].
  • Add a new excludeTags config test suite validating that excludeTags is undefined when omitted, accepts custom lists and empty arrays, and that createDefaultConfig surfaces ['#no-search-index'].
packages/core/src/__tests__/index.test.ts
packages/config/src/__tests__/index.test.ts
Document the new tag-based exclusion feature for consumers.
  • Add an "Excluding individual posts" section to the README describing the #no-search-index convention, how to override or disable it via collection.excludeTags, and how tag matching works against tag names and slugs.
  • Include a JSONC configuration example showing excludeTags usage and the disabling behavior with an empty array.
README.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • Consider precomputing the normalized excludeTags set once per GhostTypesenseManager (e.g. in the constructor) instead of rebuilding the wanted Set on every isExcludedByTag call to avoid repeated work during bulk indexing.
  • The tag normalization logic in isExcludedByTag (lower-casing, deriving hash- slugs) closely mirrors the patterns around isInternalTag; it might be worth extracting a shared helper to avoid subtle drift between these tag-matching behaviours over time.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Consider precomputing the normalized `excludeTags` set once per `GhostTypesenseManager` (e.g. in the constructor) instead of rebuilding the `wanted` `Set` on every `isExcludedByTag` call to avoid repeated work during bulk indexing.
- The tag normalization logic in `isExcludedByTag` (lower-casing, deriving `hash-` slugs) closely mirrors the patterns around `isInternalTag`; it might be worth extracting a shared helper to avoid subtle drift between these tag-matching behaviours over time.

## Individual Comments

### Comment 1
<location path="packages/core/src/index.ts" line_range="20" />
<code_context>
+ * consumer of the package — CLI, webhook handler — shares one canonical
+ * convention without each having to opt in.
+ */
+const DEFAULT_EXCLUDE_TAGS = ['#no-search-index'];
+
 export interface Post {
</code_context>
<issue_to_address>
**issue:** Default tag list is now duplicated in both core and config, which can drift over time.

This introduces a second source of truth for default exclusion tags: `DEFAULT_EXCLUDE_TAGS` here and `excludeTags: ['#no-search-index']` in `createDefaultConfig`. If only one is updated (e.g. when adding a new canonical tag), behavior will differ between callers using explicit config vs. defaults. Please centralize this (e.g. export a single constant and reuse it, or move both to a shared module) to keep the default exclusion behavior consistent.
</issue_to_address>

### Comment 2
<location path="packages/core/src/__tests__/index.test.ts" line_range="400-409" />
<code_context>
+describe('GhostTypesenseManager — tag-based exclusion (#no-search-index)', () => {
</code_context>
<issue_to_address>
**suggestion (testing):** Consider adding coverage for tag-based exclusion in the bulk indexing path (`indexAllPosts`), not just `indexPost`.

The tests for `isExcludedByTag` and `indexPost` cover the webhook/edit flow well. Since `isExcludedByTag` is also used by `indexAllPosts`, it would be valuable to add a test that bulk indexing skips `#no-search-index` (and custom `excludeTags`) posts—for example, by indexing a mix of excluded and non-excluded posts and asserting only the allowed ones reach Typesense.

Suggested implementation:

```typescript
describe('GhostTypesenseManager — tag-based exclusion (#no-search-index)', () => {
  it('skips #no-search-index posts when bulk indexing via indexAllPosts', async () => {
    const config: Config = {
      ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
      typesense: {
        nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }],
        apiKey: 'test-key'
      },
      collection: {
        name: 'test-collection',
        fields: [
          { name: 'id', type: 'string', optional: false },
          { name: 'title', type: 'string', optional: false },
          { name: 'slug', type: 'string', optional: false },
          { name: 'html', type: 'string', optional: true }
        ],
        defaultSortingField: 'id'
      }
    };

    const manager = new GhostTypesenseManager(config);

    // Mock the bulk source of posts used by indexAllPosts to return a mix of
    // excluded and non-excluded posts.
    // NOTE: adjust this to the actual client/property used by indexAllPosts.
    (manager as any).ghostAdmin = {
      posts: {
        browse: jest.fn().mockResolvedValue([
          {
            id: 'included-post',
            slug: 'included-post',
            title: 'Included Post',
            html: '<p>Included</p>',
            tags: []
          },
          {
            id: 'excluded-post',
            slug: 'excluded-post',
            title: 'Excluded Post',
            html: '<p>Excluded</p>',
            tags: [{ slug: 'no-search-index', name: '#no-search-index' }]
          }
        ])
      }
    };

    mockDocuments.upsert.mockClear();
    mockDocuments.delete.mockClear();

    await manager.indexAllPosts();

    // Only the non-excluded post should be sent to Typesense
    expect(mockDocuments.upsert).toHaveBeenCalledTimes(1);
    expect(mockDocuments.upsert).toHaveBeenCalledWith(
      expect.objectContaining({ id: 'included-post' })
    );
    expect(mockDocuments.upsert).not.toHaveBeenCalledWith(
      expect.objectContaining({ id: 'excluded-post' })
    );
    expect(mockDocuments.delete).not.toHaveBeenCalled();
  });

  it('skips custom excludeTags when bulk indexing via indexAllPosts', async () => {
    const config: Config = {
      ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
      typesense: {
        nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }],
        apiKey: 'test-key'
      },
      collection: {
        name: 'test-collection',
        fields: [
          { name: 'id', type: 'string', optional: false },
          { name: 'title', type: 'string', optional: false },
          { name: 'slug', type: 'string', optional: false },
          { name: 'html', type: 'string', optional: true }
        ],
        defaultSortingField: 'id'
      },
      excludeTags: ['#exclude-from-search']
    };

    const manager = new GhostTypesenseManager(config);

    (manager as any).ghostAdmin = {
      posts: {
        browse: jest.fn().mockResolvedValue([
          {
            id: 'included-post',
            slug: 'included-post',
            title: 'Included Post',
            html: '<p>Included</p>',
            tags: []
          },
          {
            id: 'custom-excluded-post',
            slug: 'custom-excluded-post',
            title: 'Custom Excluded Post',
            html: '<p>Excluded</p>',
            tags: [{ slug: 'exclude-from-search', name: '#exclude-from-search' }]
          }
        ])
      }
    };

    mockDocuments.upsert.mockClear();
    mockDocuments.delete.mockClear();

    await manager.indexAllPosts();

    expect(mockDocuments.upsert).toHaveBeenCalledTimes(1);
    expect(mockDocuments.upsert).toHaveBeenCalledWith(
      expect.objectContaining({ id: 'included-post' })
    );
    expect(mockDocuments.upsert).not.toHaveBeenCalledWith(
      expect.objectContaining({ id: 'custom-excluded-post' })
    );
    expect(mockDocuments.delete).not.toHaveBeenCalled();
  });

  const baseConfig: Config = {
    ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
    typesense: { nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }], apiKey: 'test-key' },
    collection: {
      name: 'test-collection',
      fields: [
        { name: 'id', type: 'string', optional: false },
        { name: 'title', type: 'string', optional: false },
        { name: 'slug', type: 'string', optional: false },
        { name: 'html', type: 'string', optional: true },

```

1. The tests above assume that `indexAllPosts` pulls posts via `(manager as any).ghostAdmin.posts.browse`. If the real implementation uses a different internal client (e.g. `ghostContentApi.posts.browse` or a different property name), adjust the mocked property accordingly so that `indexAllPosts` consumes the mocked posts.
2. Ensure `mockDocuments` is the same Jest mock used in the other tests (for `indexPost`). If its initialization lives in a shared `beforeEach`, no change is required; otherwise, you may need to expose/initialize it so these new tests can access and clear it.
3. If your `Config` type requires additional fields (e.g. search parameters, facet configs), mirror the minimal valid config from the existing tests (e.g. copy the `baseConfig` shape) to keep these new tests aligned with the rest of the suite.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread packages/core/src/index.ts Outdated
Comment on lines +400 to +409
describe('GhostTypesenseManager — tag-based exclusion (#no-search-index)', () => {
const baseConfig: Config = {
ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
typesense: { nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }], apiKey: 'test-key' },
collection: {
name: 'test-collection',
fields: [
{ name: 'id', type: 'string', optional: false },
{ name: 'title', type: 'string', optional: false },
{ name: 'slug', type: 'string', optional: false },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion (testing): Consider adding coverage for tag-based exclusion in the bulk indexing path (indexAllPosts), not just indexPost.

The tests for isExcludedByTag and indexPost cover the webhook/edit flow well. Since isExcludedByTag is also used by indexAllPosts, it would be valuable to add a test that bulk indexing skips #no-search-index (and custom excludeTags) posts—for example, by indexing a mix of excluded and non-excluded posts and asserting only the allowed ones reach Typesense.

Suggested implementation:

describe('GhostTypesenseManager — tag-based exclusion (#no-search-index)', () => {
  it('skips #no-search-index posts when bulk indexing via indexAllPosts', async () => {
    const config: Config = {
      ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
      typesense: {
        nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }],
        apiKey: 'test-key'
      },
      collection: {
        name: 'test-collection',
        fields: [
          { name: 'id', type: 'string', optional: false },
          { name: 'title', type: 'string', optional: false },
          { name: 'slug', type: 'string', optional: false },
          { name: 'html', type: 'string', optional: true }
        ],
        defaultSortingField: 'id'
      }
    };

    const manager = new GhostTypesenseManager(config);

    // Mock the bulk source of posts used by indexAllPosts to return a mix of
    // excluded and non-excluded posts.
    // NOTE: adjust this to the actual client/property used by indexAllPosts.
    (manager as any).ghostAdmin = {
      posts: {
        browse: jest.fn().mockResolvedValue([
          {
            id: 'included-post',
            slug: 'included-post',
            title: 'Included Post',
            html: '<p>Included</p>',
            tags: []
          },
          {
            id: 'excluded-post',
            slug: 'excluded-post',
            title: 'Excluded Post',
            html: '<p>Excluded</p>',
            tags: [{ slug: 'no-search-index', name: '#no-search-index' }]
          }
        ])
      }
    };

    mockDocuments.upsert.mockClear();
    mockDocuments.delete.mockClear();

    await manager.indexAllPosts();

    // Only the non-excluded post should be sent to Typesense
    expect(mockDocuments.upsert).toHaveBeenCalledTimes(1);
    expect(mockDocuments.upsert).toHaveBeenCalledWith(
      expect.objectContaining({ id: 'included-post' })
    );
    expect(mockDocuments.upsert).not.toHaveBeenCalledWith(
      expect.objectContaining({ id: 'excluded-post' })
    );
    expect(mockDocuments.delete).not.toHaveBeenCalled();
  });

  it('skips custom excludeTags when bulk indexing via indexAllPosts', async () => {
    const config: Config = {
      ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
      typesense: {
        nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }],
        apiKey: 'test-key'
      },
      collection: {
        name: 'test-collection',
        fields: [
          { name: 'id', type: 'string', optional: false },
          { name: 'title', type: 'string', optional: false },
          { name: 'slug', type: 'string', optional: false },
          { name: 'html', type: 'string', optional: true }
        ],
        defaultSortingField: 'id'
      },
      excludeTags: ['#exclude-from-search']
    };

    const manager = new GhostTypesenseManager(config);

    (manager as any).ghostAdmin = {
      posts: {
        browse: jest.fn().mockResolvedValue([
          {
            id: 'included-post',
            slug: 'included-post',
            title: 'Included Post',
            html: '<p>Included</p>',
            tags: []
          },
          {
            id: 'custom-excluded-post',
            slug: 'custom-excluded-post',
            title: 'Custom Excluded Post',
            html: '<p>Excluded</p>',
            tags: [{ slug: 'exclude-from-search', name: '#exclude-from-search' }]
          }
        ])
      }
    };

    mockDocuments.upsert.mockClear();
    mockDocuments.delete.mockClear();

    await manager.indexAllPosts();

    expect(mockDocuments.upsert).toHaveBeenCalledTimes(1);
    expect(mockDocuments.upsert).toHaveBeenCalledWith(
      expect.objectContaining({ id: 'included-post' })
    );
    expect(mockDocuments.upsert).not.toHaveBeenCalledWith(
      expect.objectContaining({ id: 'custom-excluded-post' })
    );
    expect(mockDocuments.delete).not.toHaveBeenCalled();
  });

  const baseConfig: Config = {
    ghost: { url: 'https://test.com', key: 'test-key', version: 'v5.0' },
    typesense: { nodes: [{ host: 'localhost', port: 8108, protocol: 'http' }], apiKey: 'test-key' },
    collection: {
      name: 'test-collection',
      fields: [
        { name: 'id', type: 'string', optional: false },
        { name: 'title', type: 'string', optional: false },
        { name: 'slug', type: 'string', optional: false },
        { name: 'html', type: 'string', optional: true },
  1. The tests above assume that indexAllPosts pulls posts via (manager as any).ghostAdmin.posts.browse. If the real implementation uses a different internal client (e.g. ghostContentApi.posts.browse or a different property name), adjust the mocked property accordingly so that indexAllPosts consumes the mocked posts.
  2. Ensure mockDocuments is the same Jest mock used in the other tests (for indexPost). If its initialization lives in a shared beforeEach, no change is required; otherwise, you may need to expose/initialize it so these new tests can access and clear it.
  3. If your Config type requires additional fields (e.g. search parameters, facet configs), mirror the minimal valid config from the existing tests (e.g. copy the baseConfig shape) to keep these new tests aligned with the rest of the suite.

@coderabbitai

coderabbitai Bot commented Jun 26, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@betschki, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 3 minutes and 17 seconds. Learn how PR review limits work.

To continue reviewing without waiting, enable usage-based billing in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4cf3de7c-2d02-4075-941b-a4504309ae30

📥 Commits

Reviewing files that changed from the base of the PR and between 361d957 and c8f044f.

📒 Files selected for processing (3)
  • packages/config/src/index.ts
  • packages/core/src/__tests__/index.test.ts
  • packages/core/src/index.ts
📝 Walkthrough

Walkthrough

The PR adds configurable tag-based exclusion for Ghost posts, defaults to #no-search-index, applies the exclusion in batch and webhook indexing paths, and updates documentation and tests to cover the new behavior.

Changes

Tag-based post exclusion

Layer / File(s) Summary
Config surface and docs
packages/config/src/index.ts, packages/config/src/__tests__/index.test.ts, README.md
CollectionConfigSchema adds excludeTags, createDefaultConfig seeds #no-search-index, config tests cover omitted/custom/empty arrays, and the README documents exclusion matching and override behavior.
Shared exclusion matching and batch indexing
packages/core/src/index.ts
Adds DEFAULT_EXCLUDE_TAGS and isExcludedByTag(), then filters excluded posts out of indexAllPosts() before the gated-content visibility check.
Webhook de-indexing path
packages/core/src/index.ts, packages/core/src/__tests__/index.test.ts
indexPost() deletes existing Typesense documents for excluded posts instead of upserting them, and the tests add an excluded-post fixture plus matching and webhook coverage.

Sequence Diagram(s)

sequenceDiagram
  participant GhostWebhook
  participant GhostTypesenseManager
  participant GhostAPI
  participant Typesense

  GhostWebhook->>GhostTypesenseManager: indexPost(postId)
  GhostTypesenseManager->>GhostAPI: fetch post
  GhostAPI-->>GhostTypesenseManager: post with tags
  GhostTypesenseManager->>GhostTypesenseManager: isExcludedByTag(post)
  alt post is excluded
    GhostTypesenseManager->>Typesense: documents.delete(filter_by id: postId)
  else post is not excluded
    GhostTypesenseManager->>Typesense: documents.upsert(post)
  end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Poem

I hop through posts with ears held high,
and tag the ones to pass on by.
With #no-search-index, off they go,
in batch or webhook, neat as snow.
🐰✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: excluding #no-search-index tagged posts from indexing.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/exclude-tags-no-search-index

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
packages/core/src/index.ts (1)

595-598: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Silently swallowing the de-index delete can leave excluded content searchable.

.catch(() => {}) discards every error from the de-index delete. For a content-policy escape hatch, a transient Typesense failure here means a post the publisher explicitly marked non-searchable stays in the index with no log or signal. Consider at least logging the failure (mirroring the gated branch at Line 604 if you adjust both).

🔍 Suggested logging
-      await collection.documents().delete({ filter_by: `id:${postId}` }).catch(() => {});
+      await collection.documents().delete({ filter_by: `id:${postId}` })
+        .catch((err: any) => console.error(`Failed to de-index excluded post ${postId}: ${err?.message || err}`));
       return;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/src/index.ts` around lines 595 - 598, The delete in the
`isExcludedByTag` branch of `collection.documents().delete` is swallowing all
errors, which can leave excluded posts searchable with no signal. Update this
path to handle the rejection explicitly in `indexPost`/the surrounding indexing
flow: catch the failure, log it with the same context used in the gated error
handling branch, and keep the early return so excluded content still bypasses
indexing.
packages/core/src/__tests__/index.test.ts (1)

400-470: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Good coverage of the webhook path; the bulk (indexAllPosts) filter is untested.

The new tests exercise isExcludedByTag and the indexPost webhook branch, but indexAllPosts' exclusion filter (Line 367) isn't covered — the browse mock returns no excluded post. Consider adding a bulk-path case asserting an excluded post is dropped from postsToIndex.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/core/src/__tests__/index.test.ts` around lines 400 - 470, The bulk
indexing path in GhostTypesenseManager is missing coverage for tag-based
exclusion, so add a test around indexAllPosts that uses the existing browse mock
to return an excluded post and asserts it is filtered out of postsToIndex before
indexing. Reuse the GhostTypesenseManager test setup and verify the exclusion
behavior in the bulk path, not just isExcludedByTag and indexPost.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/core/src/__tests__/index.test.ts`:
- Around line 400-470: The bulk indexing path in GhostTypesenseManager is
missing coverage for tag-based exclusion, so add a test around indexAllPosts
that uses the existing browse mock to return an excluded post and asserts it is
filtered out of postsToIndex before indexing. Reuse the GhostTypesenseManager
test setup and verify the exclusion behavior in the bulk path, not just
isExcludedByTag and indexPost.

In `@packages/core/src/index.ts`:
- Around line 595-598: The delete in the `isExcludedByTag` branch of
`collection.documents().delete` is swallowing all errors, which can leave
excluded posts searchable with no signal. Update this path to handle the
rejection explicitly in `indexPost`/the surrounding indexing flow: catch the
failure, log it with the same context used in the gated error handling branch,
and keep the early return so excluded content still bypasses indexing.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 704edc54-ee0e-4a71-a65f-340484e64c2d

📥 Commits

Reviewing files that changed from the base of the PR and between a54a3b8 and 361d957.

📒 Files selected for processing (5)
  • README.md
  • packages/config/src/__tests__/index.test.ts
  • packages/config/src/index.ts
  • packages/core/src/__tests__/index.test.ts
  • packages/core/src/index.ts

- Centralize DEFAULT_EXCLUDE_TAGS in the config package (single source of truth
  shared by core and createDefaultConfig) so the default can't drift.
- Precompute the normalized exclude-tag set once in the constructor instead of
  rebuilding it on every isExcludedByTag call during bulk indexing.
- Log instead of silently swallowing failures when de-indexing an excluded or
  gated post, so a transient delete failure isn't invisible.
- Add bulk-path coverage: indexAllPosts drops a #no-search-index post.
@betschki betschki merged commit 4c72702 into main Jun 26, 2026
7 checks passed
@betschki betschki deleted the feat/exclude-tags-no-search-index branch June 26, 2026 18:48
@betschki betschki mentioned this pull request Jun 27, 2026
betschki added a commit that referenced this pull request Jun 27, 2026
Publishes the #no-search-index tag exclusion (PR #52): posts tagged
#no-search-index (configurable via collection.excludeTags) are kept out of the
index, alongside internal-tag filtering and gated redaction. Bumps all packages
to 2.0.8 with the CHANGELOG entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant