Feat/unicode

[SuperFola]

Description

Update the standard library to add

• list:slice1
• list:first
• list:last
• list:cumulativeSum
• list:cumulativeProduct
• list:zeros
• list:ones
• string:lpad
• string:rpad
• string:ascii?
• string:first
• string:last

Also add a new builtin builtin__string:codepoints, computing the unicode codepoints of a string. This returns a list of numbers, one per codepoint.

Related to #681.

Checklist

• I have read the Contributor guide
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation if needed (on https://github.com/ArkScript-lang/website, content/docs/)
• I have added tests that prove my fix/feature is working
• New and existing tests pass locally with my changes
• I confirm that I am the author of this code and release it to the ArkScript project under the MPL-2.0 license. This contribution does not contain code from other sources, including code generated by a Large Language Model ("AI").

[github-actions]

Static analysis report

Lizard report

Listing only functions with cyclomatic complexity >= 15 or NLOC >= 100 or parameters >= 6.

Filename	Start line:end line	Function name	Parameters	NLOC	CCN
src/arkreactor/Builtins/String.cpp	47:113	fmt::formatter::parse	1	59	18

Report about files you didn't modify in this PR

Filename	Start line:end line	Function name	Parameters	NLOC	CCN
src/arkreactor/VM/VM.cpp	450:2153	Ark::VM::unsafeRun	2	1432	281
src/arkreactor/Compiler/Macros/Processor.cpp	259:634	Ark::internal::MacroProcessor::evaluate	3	344	115
src/arkreactor/Compiler/BytecodeReader.cpp	385:596	Ark::BytecodeReader::display	4	171	92
src/arkreactor/Error/Diagnostics.cpp	45:192	Ark::Diagnostics::makeContext	4	109	44
src/arkscript/main.cpp	24:383	main	2	310	43
src/arkscript/JsonCompiler.cpp	27:276	JsonCompiler::_compile	1	214	38
src/arkreactor/Compiler/NameResolution/NameResolutionPass.cpp	162:267	Ark::internal::NameResolutionPass::visitKeyword	3	85	35
src/arkreactor/Builtins/Slice.cpp	21:93	Ark::internal::Builtins::slice	1	63	34
src/arkreactor/Compiler/AST/Parser.cpp	841:932	Ark::internal::Parser::string	1	88	32
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	355:434	Ark::internal::ASTLowerer::compileListInstruction	3	64	32
src/arkreactor/Compiler/AST/Node.cpp	179:285	Ark::internal::Node::repr	0	90	30
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	853:999	Ark::internal::ASTLowerer::handleFunctionCall	3	114	30
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	212:317	Ark::internal::ASTLowerer::compileExpression	5	88	29
src/arkreactor/Compiler/Macros/Processor.cpp	107:191	Ark::internal::MacroProcessor::processNode	3	63	28
src/arkreactor/Compiler/AST/Node.cpp	287:368	Ark::internal::Node::debugPrint	1	70	26
src/arkreactor/Compiler/BytecodeReader.cpp	612:713	Ark::BytecodeReader::printInstruction	7	98	25
src/arkreactor/TypeChecker.cpp	149:236	Ark::types::generateError	6	75	25
src/arkreactor/TypeChecker.cpp	32:147	Ark::types::displayContract	6	102	24
src/arkreactor/Compiler/AST/Parser.cpp	289:417	Ark::internal::Parser::import_	1	98	23
src/arkreactor/Compiler/NameResolution/NameResolutionPass.cpp	56:160	Ark::internal::NameResolutionPass::visit	2	83	23
include/utf8.hpp	138:184	utf8::isValid	1	44	21
src/arkreactor/VM/Value/Value.cpp	77:145	Ark::Value::toString	2	54	21
src/arkreactor/VM/VM.cpp	2307:2401	Ark::VM::backtrace	3	81	20
src/arkreactor/Compiler/AST/Optimizer.cpp	35:85	Ark::internal::Optimizer::countAndPruneDeadCode	1	42	20
src/arkreactor/Compiler/NameResolution/StaticScope.cpp	70:111	Ark::internal::NamespaceScope::get	3	32	19
src/arkscript/Formatter.cpp	501:560	Formatter::formatCall	2	51	19
src/arkreactor/Compiler/AST/Optimizer.cpp	87:144	Ark::internal::Optimizer::pruneUnusedGlobalVariables	1	50	18
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	799:851	Ark::internal::ASTLowerer::handleOperator	3	42	18
src/arkscript/Formatter.cpp	186:248	Formatter::format	3	59	18
src/arkreactor/Compiler/Macros/Executors/Function.cpp	16:89	Ark::internal::FunctionExecutor::applyMacro	2	56	17
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	107:129	Ark::internal::ASTLowerer::nodeProducesOutput	1	18	17
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	597:645	Ark::internal::ASTLowerer::compileLetMutSet	4	40	17
include/Ark/Compiler/AST/Predicates.hpp	132:156	Ark::internal::IsSymbol::operator ( )	1	24	16
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	514:595	Ark::internal::ASTLowerer::compileFunction	3	59	16
src/arkscript/Formatter.cpp	298:339	Formatter::formatFunction	2	35	16
src/arkreactor/VM/VM.cpp	2165:2222	Ark::VM::throwArityError	4	47	15
src/arkreactor/Compiler/Macros/Executors/Function.cpp	99:156	Ark::internal::FunctionExecutor::unify	5	50	15
src/arkreactor/Compiler/IntermediateRepresentation/IROptimizer.cpp	22:260	Ark::internal::IROptimizer::IROptimizer	1	219	14
src/arkreactor/Error/Diagnostics.cpp	194:211	Ark::Diagnostics::helper	7	17	2

---

CppCheck report

Report files about files you didn't modify in this PR

Filename	Line	Type	Description
include/Ark/VM/VM.inl	258	style	Variable 'maybe_value_ptr' can be declared as pointer to const
src/arkreactor/Builtins/Bytecode.cpp	23	style	Parameter 'vm' can be declared as pointer to const
src/arkreactor/Builtins/IO.cpp	165	style	Consider using std::transform algorithm instead of a raw loop.
src/arkreactor/Compiler/BytecodeReader.cpp	20	performance	Variable 'm_arg_kinds' is assigned in constructor body. Consider performing initialization in initialization list.
src/arkreactor/Compiler/IntermediateRepresentation/IROptimizer.cpp	249	style	Consider using std::transform algorithm instead of a raw loop.
src/arkreactor/Compiler/IntermediateRepresentation/IROptimizer.cpp	256	style	Consider using std::transform algorithm instead of a raw loop.
src/arkreactor/Compiler/Lowerer/ASTLowerer.cpp	620	style	Consider using std::count_if algorithm instead of a raw loop.
src/arkreactor/Compiler/Lowerer/LocalsLocator.cpp	26	style	The scope of the variable 'data' can be reduced.
src/arkreactor/Compiler/NameResolution/ScopeResolver.cpp	136	style	Consider using std::find_if algorithm instead of a raw loop.
src/arkreactor/State.cpp	184	style	Variable 'bytecode_hash' is assigned a value that is never used.
src/arkreactor/VM/SharedLibrary.cpp	0	information	Too many #ifdef configurations - cppcheck only checks 12 of 13 configurations. Use --force to check all configurations.
src/arkreactor/VM/VM.cpp	0	information	Too many #ifdef configurations - cppcheck only checks 12 of 13 configurations. Use --force to check all configurations.
src/arkreactor/VM/VM.cpp	281	error	Iterators of different containers 'm_execution_contexts.emplace_back(std::make_unique())' and 'm_execution_contexts.front()' are used together.
include/Ark/VM/Value/Future.hpp	50	style	Unused private function: 'Future::deleteSelfViaVM'
src/arkreactor/VM/Value/Future.cpp	23	performance	Variable 'm_value' is assigned in constructor body. Consider performing initialization in initialization list.

[github-actions]

Super Instructions report

Some Super Instructions are under the usage threshold (17).

Super Instruction	Uses in compiled code
INCREMENT_BY_INDEX	15
INCREMENT	15
STORE_FROM	14
LT_CONST_JUMP_IF_FALSE	9
NEQ_SYM_JUMP_IF_FALSE	4
SET_VAL_TAIL	4
STORE_HEAD	2
MUL_BY	2
GT_SYM_JUMP_IF_FALSE	2
CHECK_TYPE_OF	2
NEQ_CONST_JUMP_IF_TRUE	2
SET_VAL_TAIL_BY_INDEX	1
SET_VAL_HEAD_BY_INDEX	1
MUL_SET_VAL	1
SET_VAL_HEAD	1
STORE_TAIL	1

Super Instructions over the threshold

Super Instruction	Uses in compiled code
CALL_SYMBOL	4625
LOAD_CONST_LOAD_CONST	2686
LOAD_CONST_STORE	2446
LOAD_CONST_SET_VAL	692
CALL_BUILTIN	685
INCREMENT_STORE	524
AT_SYM_SYM	516
GET_FIELD_FROM_SYMBOL_INDEX	487
CALL_BUILTIN_WITHOUT_RETURN_ADDRESS	454
STORE_LIST	365
APPEND_IN_PLACE_SYM	329
GET_FIELD_FROM_SYMBOL	301
CALL_SYMBOL_BY_INDEX	299
LT_LEN_SYM_JUMP_IF_FALSE	299
LT_SYM_JUMP_IF_FALSE	148
EQ_SYM_INDEX_JUMP_IF_TRUE	140
STORE_FROM_INDEX	107
SET_VAL_FROM_INDEX	105
SET_VAL_FROM	94
FUSED_MATH	86
AT_SYM_INDEX_SYM_INDEX	82
STORE_LEN	80
LT_CONST_JUMP_IF_TRUE	75
CALL_CURRENT_PAGE	73
DECREMENT_BY_INDEX	61
EQ_CONST_JUMP_IF_TRUE	50
GT_CONST_JUMP_IF_TRUE	46
DECREMENT_STORE	45
AT_SYM_INDEX_CONST	43
MUL_BY_INDEX	40
GT_CONST_JUMP_IF_FALSE	39
STORE_HEAD_BY_INDEX	35
STORE_TAIL_BY_INDEX	34
CHECK_TYPE_OF_BY_INDEX	27
APPEND_IN_PLACE_SYM_INDEX	22
DECREMENT	18

[codspeed-hq]

Merging this PR will degrade performance by 1.67%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 16 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	create_closure	4.7 ms	4.8 ms	-1.43%
❌	builtins	3.7 ms	3.8 ms	-1.67%

---

_{Comparing feat/unicode (1e7ff10) with dev (9a6b009)}

[coveralls]

Coverage Report for CI Build 25681357190

Coverage decreased (-0.005%) to 94.073%

Details

• Coverage decreased (-0.005%) from the base build.
• Patch coverage: 17 of 17 lines across 1 file are fully covered (100%).
• No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	10511
Covered Lines:	9888
Line Coverage:	94.07%
Coverage Strength:	711544.77 hits per line

---

💛 - Coveralls

[github-actions]

Fuzzing report

/usr/local/bin/afl-whatsup status check tool for afl-fuzz by Michal Zalewski

Summary stats

    Fuzzers alive : 0
   Dead or remote : 1 (included in stats)
   Total run time : 5 minutes, 0 seconds
      Total execs : 36 thousands
 Cumulative speed : 120 execs/sec
    Pending items : 122 faves, 1225 total
 Coverage reached : 9.68%
    Crashes saved : 0
      Hangs saved : 0

Cycles without finds : 0
Time without finds : 0

[+] Captured 44185 tuples (map size 267143, highest value 255, total values 457928871) in '/dev/null'.
[+] A coverage of 44185 edges were achieved out of 267200 existing (16.54%) with 1233 input files.