Repository navigation
Keep the functions of the BerlinMOD generator when loading the deliveries one - #74
Merged
Merged
Conversation
…ries one deliveries_datagenerator.sql builds on the functions of berlinmod_datagenerator.sql, its header says, and generates its trips with createTripC, which berlinmod_datagenerator.sql defines on the type step. deliveries_datagenerator.sql declares step again with DROP TYPE ... CASCADE, which drops createTripC, createTrip and createPath with it, so that the deliveries generator loaded after the BerlinMOD one cannot generate a trip. The type is that of berlinmod_datagenerator.sql, which deliveries no longer declares. The README and datagen.xml load berlinmod_datagenerator.sql before deliveries_datagenerator.sql. Witness: on the Brussels base map, loading berlinmod_datagenerator.sql and then deliveries_datagenerator.sql of master, deliveries_datagenerator (scaleFactor := 0.005) fails with "function createtripc(step[], timestamp with time zone, boolean, text) does not exist". Measured: with this change, the same steps generate 564 deliveries of 141 vehicles, 11,623,873 instants. Why: the trips of the deliveries are generated by the generator of BerlinMOD, on the type it defines, so the deliveries take that type and that generator as they are.
estebanzimanyi
added a commit
that referenced
this pull request
Nov 7, 2026
deliveries_datagenerator generates the whole data in one statement, which runs for longer than a statement timeout allows from the smallest of its published scale factors on: at the scale factor 0.1 on the Brussels base map its setup takes 1 min 11 s, its call of pgRouting for 37,477 paths 2 min 39 s, and its deliveries more than ten minutes more. It takes a phase, setup, paths, deliveries or finish, as berlinmod_generate does: the setup records its parameters in the table DeliveriesParameters, the paths phase runs the calls of pgRouting fromCall to toCall, of pgrBatchSize paths each, the deliveries phase generates the days fromDay to toDay, which deliveries_createDeliveries takes, numbering its deliveries after those already generated, and the finish phase builds the indexes and prints the summary. The deliveries are generated day by day, the order in which their random numbers are drawn, and only the setup sets the seed of the random numbers. deliveries_phases gives the statements after the setup, one call of pgRouting and daysPerBatch days at a time. The README shows the generation in phases. Witness: deliveries_datagenerator(scaleFactor := 0.1) on the Brussels base map is cancelled by a statement timeout of ten minutes while it generates the deliveries. Measured: at the scale factor 0.005 on the Brussels base map, with the type step of berlinmod_datagenerator.sql (#74), the generator in one call before this change, the generator in one call after it, and the generator in 8 statements of one session with one day per batch and pgRouting batches of 2000 paths generate the same 564 deliveries, whose identifiers, vehicles, days, numbers of customers and hexadecimal EWKB have the MD5 bcfee0353bcdd1a39b7b2a5efea72682 in the three cases, with the same indexes on Deliveries, Segments and Paths.
estebanzimanyi
added a commit
that referenced
this pull request
Nov 8, 2026
) * Generate the BerlinMOD data in phases of one session berlinmod_generate generates the whole data in one statement, which runs for longer than a statement timeout allows at a large scale factor: at the scale factor 0.1 on the Brussels base map it runs for 5 min 2 s, 17 s of setup, 50 s of pgRouting and 3 min 54 s of trips, and the trips grow with the scale factor. berlinmod_generate takes a phase, setup, paths, trips or finish, all by default: the setup records its parameters in the table GenerationParameters, which the later phases read, the paths phase runs the calls of pgRouting fromCall to toCall, the trips phase generates the vehicles fromVehicle to toVehicle, which berlinmod_createTrips takes, and the finish phase builds the indexes of Trips and prints the summary. berlinmod_phases gives the statements of the phases after the setup, one call of pgRouting and vehiclesPerBatch vehicles at a time, which the command \gexec of psql runs in the session of the setup, so that the random numbers are drawn in the order of one call. The message that ends a call of pgRouting printed the variable i, which the function does not declare, so that a generation of more than one call failed with messages medium or verbose; it prints c. The README shows the generation in phases. Witness: berlinmod_generate(scaleFactor := 0.1) on the Brussels base map runs as one statement of 5 min 2 s, and the trips of the scale factors 0.5 and 1, 81,584 and 157,565 against the 18,910 of 0.1, run longer than ten minutes. Measured: at the scale factor 0.02 on the Brussels base map, the generator of master in one call, this generator in one call, and this generator in 39 statements of one session with 8 vehicles per batch generate the same 4641 trips, whose vehicles, days, numbers and hexadecimal EWKB have the MD5 3faac183bad2d041181c4097152cb583 in the three cases, with the same indexes on Trips and Paths. * Generate the deliveries in phases of one session deliveries_datagenerator generates the whole data in one statement, which runs for longer than a statement timeout allows from the smallest of its published scale factors on: at the scale factor 0.1 on the Brussels base map its setup takes 1 min 11 s, its call of pgRouting for 37,477 paths 2 min 39 s, and its deliveries more than ten minutes more. It takes a phase, setup, paths, deliveries or finish, as berlinmod_generate does: the setup records its parameters in the table DeliveriesParameters, the paths phase runs the calls of pgRouting fromCall to toCall, of pgrBatchSize paths each, the deliveries phase generates the days fromDay to toDay, which deliveries_createDeliveries takes, numbering its deliveries after those already generated, and the finish phase builds the indexes and prints the summary. The deliveries are generated day by day, the order in which their random numbers are drawn, and only the setup sets the seed of the random numbers. deliveries_phases gives the statements after the setup, one call of pgRouting and daysPerBatch days at a time. The README shows the generation in phases. Witness: deliveries_datagenerator(scaleFactor := 0.1) on the Brussels base map is cancelled by a statement timeout of ten minutes while it generates the deliveries. Measured: at the scale factor 0.005 on the Brussels base map, with the type step of berlinmod_datagenerator.sql (#74), the generator in one call before this change, the generator in one call after it, and the generator in 8 statements of one session with one day per batch and pgRouting batches of 2000 paths generate the same 564 deliveries, whose identifiers, vehicles, days, numbers of customers and hexadecimal EWKB have the MD5 bcfee0353bcdd1a39b7b2a5efea72682 in the three cases, with the same indexes on Deliveries, Segments and Paths.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
deliveries_datagenerator.sql builds on the functions of
berlinmod_datagenerator.sql, its header says, and generates its trips with
createTripC, which berlinmod_datagenerator.sql defines on the type step.
deliveries_datagenerator.sql declares step again with DROP TYPE ... CASCADE,
which drops createTripC, createTrip and createPath with it, so that the
deliveries generator loaded after the BerlinMOD one cannot generate a trip.
The type is that of berlinmod_datagenerator.sql, which deliveries no longer
declares. The README and datagen.xml load berlinmod_datagenerator.sql before
deliveries_datagenerator.sql.
Witness: on the Brussels base map, loading berlinmod_datagenerator.sql and
then deliveries_datagenerator.sql of master, deliveries_datagenerator
(scaleFactor := 0.005) fails with "function createtripc(step[], timestamp
with time zone, boolean, text) does not exist".
Measured: with this change, the same steps generate 564 deliveries of 141
vehicles, 11,623,873 instants.
Why: the trips of the deliveries are generated by the generator of
BerlinMOD, on the type it defines, so the deliveries take that type and
that generator as they are.