From 84da8c1329497ae226277f6b2c2e915df58a7b57 Mon Sep 17 00:00:00 2001 From: Ben Halpern Date: Mon, 13 Jul 2026 11:35:59 -0400 Subject: [PATCH] docs: add guide on working with concepts API --- docs/api.md | 3 + docs/concepts.md | 200 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 203 insertions(+) create mode 100644 docs/concepts.md diff --git a/docs/api.md b/docs/api.md index 0a9c9b3..33560bd 100644 --- a/docs/api.md +++ b/docs/api.md @@ -28,3 +28,6 @@ to use an `accept` header, and many endpoints (but not all) require `api-key` he The `accept` header must be set to `application/vnd.forem.api-v1+json`. The `api-key` header must be set to the API key for the user. This API key can be generated by the user via their settings page in Forem. + +For a detailed guide on using and managing semantic concepts via the API, see [Working with Concepts](/concepts). + diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 0000000..754ddfd --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,200 @@ +--- +title: Working with Concepts +slug: /concepts +--- + +# Working with Concepts + +Concepts in Forem allow you to group and classify content semantically using advanced machine learning models (specifically `gemini-embedding-2`). Instead of relying solely on traditional tagging or exact keyword matching, concepts define a "semantic anchor" using a name and description. Forem automatically generates embeddings for these anchors and maps articles to them when the semantic similarity exceeds a specified threshold. + +This guide is designed for curators, creators, and users who have been granted access to manage specific concepts on a Forem platform (e.g., [DEV.to](https://dev.to)), as well as Forem administrators who manage the concepts ecosystem. + +--- + +## Authorization & Roles + +Access to the Concepts API is determined by your user role and explicit permissions: + +1. **Users with Concept Access (Curators / Specific Owners)**: + - If a Forem administrator has granted you access to a concept, that concept is considered one of your **accessible concepts**. + - You can list your accessible concepts, retrieve their details, fetch articles mapped to them, and **update** their metadata (e.g., description, similarity threshold, and score). + +2. **Super Administrators**: + - Have full system-wide access. + - Can **create** new concepts globally, **delete** concepts, and trigger **historical backfills (lookbacks)** to categorize older articles. + +:::info API Headers +All requests to the Forem API require the following headers: +- `Accept: application/vnd.forem.api-v1+json` +- `api-key: YOUR_API_KEY` (Generate this from your settings page on DEV.to) +::: + +--- + +## Accessing Concepts + +These endpoints allow you to query, search, and retrieve concepts that you have permission to access. + +### 1. List All Accessible Concepts +Retrieves a paginated list of concepts that you have been granted access to. + +* **HTTP Method:** `GET` +* **Path:** `/api/concepts` +* **Query Parameters:** + - `page` (integer, optional): The page number for pagination. Default is `1`. + - `per_page` (integer, optional): The number of items to return per page. Default is `10`. + - `days` (integer, optional): The window of days used to calculate popularity scores for the concepts (e.g., `7` or `14`). + +#### Example Request +```bash +curl -X GET "https://dev.to/api/concepts?page=1&per_page=10&days=7" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_API_KEY" +``` + +--- + +### 2. Retrieve Details of a Single Concept +Retrieves detailed information for a specific concept, including its similarity thresholds, description, and popularity metrics. + +* **HTTP Method:** `GET` +* **Path:** `/api/concepts/{id}` +* **Path Parameters:** + - `id` (integer, required): The ID of the concept. +* **Query Parameters:** + - `days` (integer, optional): The window of days to calculate the popularity score for this concept. + +#### Example Request +```bash +curl -X GET "https://dev.to/api/concepts/123?days=14" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_API_KEY" +``` + +--- + +### 3. Retrieve Articles Mapped to a Concept +Retrieves a paginated list of articles that have been classified under a specific concept. + +* **HTTP Method:** `GET` +* **Path:** `/api/concepts/{id}/articles` +* **Path Parameters:** + - `id` (integer, required): The ID of the concept. +* **Query Parameters:** + - `page` (integer, optional): Page number for pagination. Default is `1`. + - `per_page` (integer, optional): Number of articles to return per page. Default is `20`. + - `sort` (string, optional): How to order the resulting articles. + - `-similarity` (default): Orders articles by semantic relevance (cosine distance to the concept's anchor embedding), showing the most relevant articles first. + - `-published_at`: Orders articles chronologically, showing the most recently published articles first. + +#### Example Request +```bash +curl -X GET "https://dev.to/api/concepts/123/articles?sort=-similarity&page=1&per_page=20" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_API_KEY" +``` + +--- + +## Modifying & Managing Concepts + +Depending on your permissions, you can create, modify, backfill, or delete concepts. + +### 1. Update a Concept's Metadata +If you have access to a concept (or if you are a Super Admin), you can update its metadata to refine classification rules. + +* **HTTP Method:** `PATCH` +* **Path:** `/api/concepts/{id}` +* **Path Parameters:** + - `id` (integer, required): The ID of the concept to update. +* **Request Body (JSON):** + - `concept` (object, required): + - `description` (string, optional): Updates the semantic description used to calculate the anchor embedding. Modifying this will recompute the concept's embedding, which may change which articles are classified under this concept. + - `similarity_threshold` (number, optional): A value between `0.0` and `1.0` (typically `0.7` to `0.9`). Determines how closely an article's embedding must match the concept's anchor embedding to be classified under it. A higher threshold makes classification stricter (fewer, highly relevant articles), while a lower threshold is more permissive (more articles, potential for noise). + - `score` (number, optional): A weight/priority score assigned to the concept. + +#### Example Request +```bash +curl -X PATCH "https://dev.to/api/concepts/123" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "concept": { + "description": "Updated machine learning and deep neural networks definition.", + "similarity_threshold": 0.80 + } + }' +``` + +--- + +### 2. Create a New Concept (Super Admin Only) +Super Admins can create new global concepts. When created, Forem automatically generates a semantic anchor embedding using `gemini-embedding-2` based on the concept's initial description. + +* **HTTP Method:** `POST` +* **Path:** `/api/admin/concepts` +* **Request Body (JSON):** + - `concept` (object, required): + - `name` (string, required): The display name of the concept. + - `description` (string, required): The detailed description used to generate the semantic embedding. Be descriptive to ensure high-quality matching. + - `similarity_threshold` (number, required): The threshold for matching (e.g., `0.75`). + - `score` (number, optional): The priority score/weight (e.g., `1.0`). + +#### Example Request +```bash +curl -X POST "https://dev.to/api/admin/concepts" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_SUPER_ADMIN_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "concept": { + "name": "Machine Learning", + "description": "Artificial intelligence, deep learning, training neural networks, transformers, and LLMs.", + "similarity_threshold": 0.75, + "score": 1.0 + } + }' +``` + +--- + +### 3. Trigger Historical Backfill / Lookback (Super Admin Only) +When a concept is newly created or significantly updated, newly published articles will be classified automatically. However, existing articles are not automatically re-evaluated unless you trigger a historical lookback. + +This endpoint queues a background worker to scan articles published within the last `N` days and evaluate them against the concept's anchor embedding. + +* **HTTP Method:** `POST` +* **Path:** `/api/admin/concepts/{id}/trigger_lookback` +* **Path Parameters:** + - `id` (integer, required): The ID of the concept. +* **Request Body (JSON):** + - `days` (integer, required): The number of historical days to scan (e.g., `30` to backfill the last month of articles). + +#### Example Request +```bash +curl -X POST "https://dev.to/api/admin/concepts/123/trigger_lookback" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_SUPER_ADMIN_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "days": 30 + }' +``` + +--- + +### 4. Delete a Concept (Super Admin Only) +Permanently deletes the concept anchor and removes all associated article classifications. This action is irreversible. + +* **HTTP Method:** `DELETE` +* **Path:** `/api/admin/concepts/{id}` +* **Path Parameters:** + - `id` (integer, required): The ID of the concept to delete. + +#### Example Request +```bash +curl -X DELETE "https://dev.to/api/admin/concepts/123" \ + -H "Accept: application/vnd.forem.api-v1+json" \ + -H "api-key: YOUR_SUPER_ADMIN_API_KEY" +```