ShExStatements

ShExStatements allows users to generate Shape Expressions (ShEx) from simple CSV statements, CSV files, and spreadsheets. It can be used from the command line, via REST API, or through a modern web interface.

Python compatibility

• Core CSV/Spreadsheet to ShEx conversion supports modern Python versions including Python 3.13.
• CI runs on Python 3.12, 3.13, plus 3.14-dev (allowed to fail) to detect future breakages early.

Ways to use ShExStatements

ShExStatements currently supports three primary usage modes:

1. WASM runtime in the browser (static frontend, no backend required)
2. Docker runtime (React frontend + FastAPI backend)
3. Python runtime (CLI and legacy Flask interface)

Quick start

1) Using Python (CLI)

Set up a virtual environment and install shexstatements:

$ python3 -m venv .venv
$ source ./.venv/bin/activate
$ pip3 install shexstatements

Run the following command with an example CSV file. The file contains an example description of a language on Wikidata. This file uses comma as a delimiter to separate the values.

$ shexstatements.sh examples/language.csv

2) Using Docker (Frontend + Backend)

Run the containerized stack:

cd docker
docker compose up

This starts:

• Frontend: http://localhost:3000
• Backend API: http://localhost:8000
• Swagger/OpenAPI docs: http://localhost:8000/docs

For development mode with hot reloading:

cd docker
docker compose -f docker-compose.yml -f docker-compose.dev.yml up

Build from source

Terminal

Clone the ShExStatements repository.

$ git clone https://github.com/johnsamuelwrites/ShExStatements.git

Go to ShExStatements directory.

$ cd ShExStatements

Install modules required by ShExStatements (here: installing into a virtual environment).

$ python3 -m venv .venv
$ source ./.venv/bin/activate
$ pip3 install .

Run the following command with an example CSV file. The file contains an example description of a language on Wikidata. This file uses comma as a delimiter to separate the values.

$ ./shexstatements.sh examples/language.csv

CSV file can use delimiters like ;. Take for example, the following command works with a file using semi-colon as a delimiter.

$ ./shexstatements.sh examples/languagedelimsemicolon.csv --delim ";"

But sometimes, users may like to specify the header. In that case, they can make use of -s or --skipheader to tell the generator to skip the header (first line of CSV).

$ ./shexstatements.sh --skipheader examples/header/languageheader.csv

It is also possible to work with Spreadsheet files like .ods, .xls or .xlsx.

$ shexstatements.sh examples/language.ods

$ shexstatements.sh examples/language.xls

$ shexstatements.sh examples/language.xlsx

In all the above cases, the shape expression generated by ShExStatements will look like

PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX wd: <http://www.wikidata.org/entity/>
PREFIX wdt: <http://www.wikidata.org/prop/direct/>
start = @<language>
<language> {
  wdt:P31 [ wd:Q34770  ] ;# instance of a language
  wdt:P1705 LITERAL ;# native name
  wdt:P17 .+ ;# spoken in country
  wdt:P2989 .+ ;# grammatical cases
  wdt:P282 .+ ;# writing system
  wdt:P1098 .+ ;# speakers
  wdt:P1999 .* ;# UNESCO language status
  wdt:P2341 .+ ;# indigenous to
}

It's also possible to use application profiles of the following form

Entity_name,Property,Property_label,Mand,Repeat,Value,Value_type,Annotation

and Shape expressions can be generated using the following form

$ ./shexstatements.sh -ap --skipheader examples/languageap.csv

Objectives

• Easily generate shape expressions (ShEx) from CSV files and Spreadsheets
• Simple syntax

Documentation and examples

A detailed documentation is available here, with example CSV files in the examples folder.

Test cases and coverage

All the test cases can be run in the following manner

$ python3 -m tests.tests

Code coverage report can also be generated by running the unit tests using the coverage tool.

$ coverage run --source=shexstatements -m unittest tests.tests
$ coverage report -m

Web Interface

Modern Web Interface (v1.0+)

ShExStatements now includes a modern, feature-rich web interface built with React and TypeScript.

Using Docker (recommended):

cd docker
docker compose up

Access the interface at http://localhost:3000

Features:

• Split-pane editor with Monaco Editor (VS Code-like experience)
• Syntax highlighting for ShExStatements and ShEx output
• Dark mode support
• File upload support (CSV, ODS, XLS, XLSX)
• Multiple delimiter options (comma, pipe, semicolon)
• Real-time error display
• Copy output to clipboard
• Runtime selector (Auto, API, WASM)

Static GitHub Pages (WASM)

The frontend can run conversion directly in the browser using Python-on-WASM (Pyodide), so it can be deployed as a static site on GitHub Pages.

1. Enable GitHub Pages in repository settings (source: GitHub Actions).
2. Push to main or master.
3. The workflow .github/workflows/pages.yml builds and deploys the frontend with VITE_RUNTIME_MODE=wasm.

In WASM runtime:

• CSV conversion to ShEx is supported in-browser.
• Spreadsheet uploads (.xlsx, .xls, .ods) are also supported in-browser.
• Pyodide dynamically installs Python dependencies (shexstatements, ply, and spreadsheet libraries) in the browser runtime.

Legacy Web Interface

The original Flask-based interface is still available:

$ python3 -m venv .venv
$ source ./.venv/bin/activate
$ pip3 install .
$ ./shexstatements.sh -r

Check the URL http://127.0.0.1:5000/

API

ShExStatements provides a REST API for programmatic access.

Modern API (v1.0+)

The new FastAPI-based API provides:

• OpenAPI/Swagger documentation at http://localhost:8000/docs
• Async request handling
• Structured JSON responses with error details

Convert endpoint:

curl -X POST http://localhost:8000/api/v1/convert \
  -H "Content-Type: application/json" \
  -d '{"content": "@shape|prop|value", "delimiter": "|", "output_format": "shex"}'

API documentation

Detailed API documentation (modern v1 API and legacy compatibility notes) is available here.

Deployment Modes

• Standalone Python application: CLI + legacy Flask UI (./shexstatements.sh).
• Docker application: React frontend + FastAPI backend (docker compose up).
• Static GitHub Pages frontend: WASM runtime (no backend required for CSV-to-ShEx).

Demonstration

Online demonstrations are also available:

• https://shexstatements.toolforge.org/
• https://tools.wmflabs.org/shexstatements/

Author

• John Samuel
• Contributors

Conference Proceedings

• ShExStatements: Simplifying Shape Expressions for Wikidata , John Samuel, Wiki Workshop 2021 (held at The Web Conference 2021), 14 April 2021 (PDF, Slides)

Acknowledgements

• Wikidata Community

Archives and Releases

• Zenodo
• Software Heritage
• Release Notes
• Changelog

Licence

All code are released under GPLv3+ licence. The associated documentation and other content are released under CC-BY-SA.