| Subject: | bulk load takes too long due to excessive fsync() |
The first use of ORDB::Unihan creates the SQLite database in the
background. Every few kilobytes, the database syncs the changes and
journal to disk. This is exceedingly slow. No one has the patience for
a three hour build time. With the following patch, creating the
database only takes a minute.
diff --git a/lib/ORDB/Unihan.pm b/lib/ORDB/Unihan.pm
index 39c8af2..650cc9f 100644
--- a/lib/ORDB/Unihan.pm
+++ b/lib/ORDB/Unihan.pm
@@ -150,6 +150,10 @@ sub import {
PrintError => 1,
}
);
+ $dbh->do('PRAGMA synchronous=OFF');
+ $dbh->do('PRAGMA count_changes=OFF');
+ $dbh->do('PRAGMA journal_mode=MEMORY');
+ $dbh->do('PRAGMA temp_store=MEMORY');
$dbh->do(<<'SQL');
CREATE TABLE unihan (
"hex" CHAR(5) NOT NULL,