Add bidirectional creator↔collection links via EAD authfilenumber and EAC-CPF descriptiveNote

[Copilot]

• Part 1: Collections → Creators - Add creator_arclight_ids_ssim to EAD
  • In task_resource(): calls new add_creator_ids_to_ead() helper
  • New example_traject_config_ead_extra.rb shows how to index arcuit:creator_id attributes
• Part 2: Creators → Collections - Add collection_arclight_ids_ssim to EAC-CPF
  • In task_agent(): calls new add_collection_links_to_eac_cpf() helper
  • example_traject_config_eac_cpf.rb: new to_field 'collection_arclight_ids_ssim'
• Refactoring based on code review
  • Fix idempotent check in add_collection_links_to_eac_cpf() to look for full pattern
  • Create XmlTransformService with XML manipulation methods
  • Create AgentService for agent data fetching
  • Add tests for both services (28 tests, all passing)
  • Refactor task_resource() to use inject_collection_metadata() service
  • Refactor task_agent() to use validate_eac_cpf_xml() service
  • Refactor get_creator_bioghist() to use AgentService and XmlTransformService
  • Delegate XML transformations to services (add_creator_ids_to_ead, add_collection_links_to_eac_cpf)
  • Use relative imports for services
  • Add package documentation
  • Refactor XML manipulation to use ElementTree parser instead of string manipulation
  • Make all XML transformations namespace-aware for proper handling of EAD and EAC-CPF namespaces
  • Use ElementTree for building bioghist elements to ensure proper XML escaping
  • Restore validate_eac_cpf_xml method with namespace fallback handling
  • Properly escape XML special characters in bioghist paragraph content
  • Address code review feedback (docstrings, validation, test formatting, namespace comments)

Original prompt

Background

We are improving how creator records and collections are linked in ArcLight/Solr. Previously, we inferred creator information from bioghist statements in the first search result, which was clunky and error-prone when multiple creators were involved or when collections had custom bioghist notes.

Goal

Create explicit bidirectional links between creators and collections in Solr by:

1. Adding creator_arclight_ids_ssim field to collection records containing array of creator IDs
2. Adding collection_arclight_ids_ssim field to creator records containing array of collection EAD IDs

Implementation Requirements

Part 1: Collections → Creators Linkage

Location: arcflow/main.py in task_resource() method

Approach:

1. After fetching the resource with resolved linked_agents (line ~235-239)
2. Extract agent refs where role='creator'
3. Convert ArchivesSpace agent URIs (e.g., /agents/corporate_entities/123) to creator IDs (e.g., creator_corporate_entities_123)
4. Parse the EAD XML and find <origination> elements
5. Add authfilenumber attribute to the name elements (<corpname>, <persname>, <famname>) with the corresponding creator ID
6. This keeps the EAD standards-compliant since authfilenumber is a valid EAD attribute for authority file identifiers

Traject Config Update:

• File: arclight_dir/lib/arcuit/traject/ead_extra_config.rb (or wherever EAD collection indexing happens)
• Extract authfilenumber attributes from origination elements
• Index as creator_arclight_ids_ssim (string, stored, indexed, multivalued)

Part 2: Creators → Collections Linkage

Location: arcflow/main.py in task_agent() method

Approach:

1. After fetching EAC-CPF XML (line ~880)
2. Parse the XML to find <resourceRelation resourceRelationType="creatorOf"> elements
3. Extract xlink:href attribute values (ArchivesSpace resource URIs like https://archivesspace-stage.library.illinois.edu/repositories/2/resources/200)
4. For each URI:
   • Extract repo_id and resource_id
   • Use ASnake client to fetch the resource: self.client.get('/repositories/{repo_id}/resources/{resource_id}').json()
   • Get the ead_id from the response
   • Handle errors: if resource can't be fetched (deleted, unpublished, etc.), log warning and skip that collection
5. Add <descriptiveNote><p>ead_id:{ead_id}</p></descriptiveNote> child element to each <resourceRelation> element (standard EAC-CPF element)
6. Save the modified EAC-CPF XML

Traject Config Update:

• File: example_traject_config_eac_cpf.rb
• Extract ead_ids from <resourceRelation><descriptiveNote> elements
• Index as collection_arclight_ids_ssim (string, stored, indexed, multivalued)

Technical Details

Creator ID Format

Use the existing pattern: creator_{agent_type}_{id}

• Entity types: corporate_entities, people, families
• Example: creator_corporate_entities_123

Collection ID Format

Use the ead_id field from ArchivesSpace resources

• Example: ALA.9.5.16

Helper Functions

Reuse the pattern from the existing traject config:

• aspace_uri_to_solr_id(uri) - converts ArchivesSpace URIs to creator IDs
• Create Python equivalent for use in main.py

Error Handling

• If fetching a resource fails (404, permission error, etc.):
  • Log a warning with the resource URI
  • Skip that specific collection
  • Continue processing the creator and other collections

Example Data Flow

EAC-CPF Input (from ArchivesSpace):

<resourceRelation resourceRelationType="creatorOf" 
                  xlink:href="https://archivesspace-stage.library.illinois.edu/repositories/2/resources/200">
  <relationEntry>Certified Public Library Administrator (CPLA)</relationEntry>
</resourceRelation>

Modified EAC-CPF (after processing):

<resourceRelation resourceRelationType="creatorOf" 
                  xlink:href="https://archivesspace-stage.library.illinois.edu/repositories/2/resources/200">
  <relationEntry>Certified Public Library Administrator (CPLA)</relationEntry>
  <descriptiveNote>
    <p>ead_id:ALA.9.5.16</p>
  </descriptiveNote>
</resourceRelation>

EAD Input (from ArchivesSpace):

<origination label="Creator">
  <corpname source="lcnaf">ALA Allied Professional Association</corpname>
</origination>

Modified EAD (after processing):

<origination label="Creator">
  <corpname source="lcnaf" authfilenumber="creator_corporate_entities_123">ALA Allied Professional Association</corpname>
</origination>

Branch Strategy

• Base branch: index_creators (contains all the creator indexing infrastructure)
• Target branch: index_creators (will be merged to main separately)

Testing Considerations

• Verify that collections with multiple creators get multiple IDs in array
• Verify that creators linked to multiple collections get multiple EAD IDs in array
• Verify error handling when resources are unpublished/deleted
• Verify that modified...

This pull request was created from Copilot chat.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[alexdryden]

@copilot we are running into a lot of digital objects in the EAC_CPF, like so:

      <resourceRelation resourceRelationType="creatorOf" xlink:href="https://archivesspace-stage.library.illinois.edu/repositories/7/digital_objects/627" xlink:type="simple">
        <relationEntry>Jack Sherman Baker Papers (Digital Surrogates and Born Digital Records)</relationEntry>
      </resourceRelation>

This is creating a lot of needless network traffic as we hit that endpoint and can't parse the result. How about we verify that the url is a resource first so we can lighten the network load. For reference, what we are after are these:

      <resourceRelation resourceRelationType="creatorOf" xlink:href="https://archivesspace-stage.library.illinois.edu/repositories/7/resources/1216" xlink:type="simple">
        <relationEntry>Jack Sherman Baker Papers</relationEntry>

So, essentially we need to check that it has the form: repositories/{number}/resources/{number}

[Copilot]

@copilot we are running into a lot of digital objects in the EAC_CPF, like so:

      <resourceRelation resourceRelationType="creatorOf" xlink:href="https://archivesspace-stage....

Updated in ae3b79e to validate URLs match the pattern repositories/{number}/resources/{number} before attempting to fetch them. Digital objects and other non-resource URLs are now skipped silently without generating warnings or network traffic.

[alexdryden]

This is too broad, is a valid element and might just be in the eac_cpf. You need to look for the full pattern of what we are making--keep in mind that the ead_id can take a variety of forms.

[alexdryden]

@copilot see feedback for changes. if you have any questions or something seems unclear, stop and ask before you proceed.

[alexdryden]

This is too broad, is a valid element and might just be in the eac_cpf. You need to look for the full pattern of what we are making--keep in mind that the ead_id can take a variety of forms.

[Copilot]

Fixed in 1b35516. Changed the check to match the specific pattern we create: <descriptiveNote>\s*<p>ead_id:[^<]+</p>\s*</descriptiveNote> instead of just looking for any <descriptiveNote> tag.

[alexdryden]

@copilot apply changes based on the comments in this thread that are not already resolved. Note that some of them that relate to the namespaceing are partially resolved.

[Copilot]

@copilot apply changes based on the comments in this thread that are not already resolved. Note that some of them that re...

Applied namespace-aware fixes in commit 0463690:

1. inject_collection_metadata: Now creates recordgroup, subgroup, and bioghist elements in the same namespace as the parent EAD document
2. add_collection_links_to_eac_cpf: Detects EAC-CPF namespace and creates descriptiveNote and p elements with proper namespace
3. build_bioghist_element: Refactored to use ElementTree for proper XML construction and escaping instead of string concatenation
4. validate_eac_cpf_xml: Restored method with namespace fallback handling for xlink attributes
5. Tests: Updated to handle namespaced XML output from ElementTree serialization

All 17 tests passing.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 10 changed files in this pull request and generated 5 comments.

[Copilot]

The _extract_paragraphs docstring says it returns strings “wrapped in

tags”, but the implementation returns plain text lines and relies on build_bioghist_element to wrap them. Updating the docstring will prevent consumers from double-wrapping or misusing the returned data.

Suggested change

	List of paragraph strings wrapped in <p> tags
	List of plain text paragraph strings (not wrapped in <p> tags)

[Copilot]

The PR description mentions refactoring task_agent() to use a validate_eac_cpf_xml() service, but the updated code no longer validates the EAC-CPF document before writing it (it only attempts a parse inside add_collection_links_to_eac_cpf). If validation is still required, consider calling validate_eac_cpf_xml() here and aborting on invalid XML, or update the PR description if validation was intentionally removed.

[Copilot]

Serializing with ET.tostring() after parsing will typically rewrite the original namespace prefixes (e.g., default EAD/EAC namespaces often become ns0:) and will drop the XML declaration. If downstream tooling expects ArchivesSpace’s original namespace/prefix layout or the <?xml ...?> prolog, consider registering the document’s default namespace before serializing and/or writing with ElementTree.write(..., xml_declaration=True, encoding='utf-8') to preserve the prolog.

[Copilot]

This test method’s body is indented more than necessary (extra level under the def), which is easy to miss in reviews and makes future edits error-prone. Align the docstring and body to a single indentation level inside the method (4 spaces beyond the def).

[alexdryden]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied changes in commit 82c2e23:

1. Updated _extract_paragraphs() docstring to clarify it returns plain text (not wrapped in <p> tags)
2. Added validate_eac_cpf_xml() call in task_agent() to validate EAC-CPF XML before processing
3. Fixed over-indentation in test_multiple_creators_with_namespace() test method
4. Added comment documenting ElementTree serialization behavior regarding namespace prefix rewriting and XML declaration