Skip to content

Add new --append option to IndexBuilder to support adding new items to an existing index #58

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MaJaHa95 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add new --append option to IndexBuilder to support adding new items to an existing index #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
75f2759
Add new --append option to IndexBuilder to support adding new items t…
MaJaHa95 May 30, 2019
39e53e7
Correct tabs to spaces
MaJaHa95 May 30, 2019
10ec610
Update GettingStart.md
MaJaHa95 May 30, 2019
a5610ac
Fix shared pointer assignment
MaJaHa95 May 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AnnService/inc/IndexBuilder/Options.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ class BuilderOptions : public Helper::ArgumentsParser

std::string m_vectorDelimiter;

bool m_append;

SPTAG::VectorValueType m_inputValueType;

std::string m_inputFiles;
Expand Down
1 change: 1 addition & 0 deletions AnnService/src/IndexBuilder/Options.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ BuilderOptions::BuilderOptions()
AddRequiredOption(m_outputFolder, "-o", "--outputfolder", "Output folder.");
AddRequiredOption(m_indexAlgoType, "-a", "--algo", "Index Algorithm type.");
AddOptionalOption(m_builderConfigFile, "-c", "--config", "Config file for builder.");
AddOptionalOption(m_append, "-p", "--append", "Append to existing index.");
}


Expand Down
26 changes: 19 additions & 7 deletions AnnService/src/IndexBuilder/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,9 @@ int main(int argc, char* argv[])
indexBuilder->SetParameter(iter.first.c_str(), iter.second.c_str());
}

ErrorCode code;
std::shared_ptr<VectorSet> p_vectorSet = nullptr;
std::shared_ptr<MetadataSet> p_metaSet = nullptr;

if (options->m_inputFiles.find("BIN:") == 0) {
std::vector<std::string> files = SPTAG::Helper::StrUtils::SplitString(options->m_inputFiles.substr(4), ",");
std::ifstream inputStream(files[0], std::ifstream::binary);
Expand All @@ -71,14 +73,12 @@ int main(int argc, char* argv[])
char* vecBuf = reinterpret_cast<char*>(vectorSet.Data());
inputStream.read(vecBuf, totalRecordVectorBytes);
inputStream.close();
std::shared_ptr<VectorSet> p_vectorSet(new BasicVectorSet(vectorSet, options->m_inputValueType, col, row));

p_vectorSet.reset(new BasicVectorSet(vectorSet, options->m_inputValueType, col, row));

std::shared_ptr<MetadataSet> p_metaSet = nullptr;
if (files.size() >= 3) {
p_metaSet.reset(new FileMetadataSet(files[1], files[2]));
}
code = indexBuilder->BuildIndex(p_vectorSet, p_metaSet);
indexBuilder->SaveIndex(options->m_outputFolder);
}
else {
auto vectorReader = IndexBuilder::VectorSetReader::CreateInstance(options);
Expand All @@ -87,9 +87,21 @@ int main(int argc, char* argv[])
fprintf(stderr, "Failed to read input file.\n");
exit(1);
}
code = indexBuilder->BuildIndex(vectorReader->GetVectorSet(), vectorReader->GetMetadataSet());
indexBuilder->SaveIndex(options->m_outputFolder);

p_vectorSet = vectorReader->GetVectorSet();
Copy link
Copy Markdown
Author

@MaJaHa95 MaJaHa95 May 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there an issue with using the out-of-scope vectorReader here? I wasn't sure about the lifetime of it, whether there could be an issue. Again, new to all this.

p_metaSet = vectorReader->GetMetadataSet();
}

ErrorCode code;
std::shared_ptr<SPTAG::VectorIndex> vecIndex;
if (options->m_append && ErrorCode::Success == indexBuilder->LoadIndex(options->m_outputFolder, vecIndex) && nullptr != vecIndex) {
code = vecIndex->AddIndex(p_vectorSet, p_metaSet);
indexBuilder = vecIndex;
}
else {
code = indexBuilder->BuildIndex(p_vectorSet, p_metaSet);
}
indexBuilder->SaveIndex(options->m_outputFolder);

if (ErrorCode::Success != code)
{
Expand Down
1 change: 1 addition & 0 deletions docs/GettingStart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
-o, --outputfolder <value> Output folder, required.
-a, --algo <value> Index Algorithm type (e.g. BKT, KDT), required.

-p, --append <true|false> Indicate whether the data should be appended to an existing index, if one exists.
-t, --thread <value> Thread Number, default is 32.
--delimiter <value> Vector delimiter, default is |.
Index.<ArgName>=<ArgValue> Set the algorithm parameter ArgName with value ArgValue.
Expand Down