clojure java.jdbc_jdbc - clojure.java.jdbc /查询大结果集懒惰 - SO中文参考 - www.soinside.com...

我通过将clojure.java.jdbc放在我的project.clj依赖项列表中来获取[org.clojure/java.jdbc "0.3.0-beta1"]最近的修复。这个增强/纠正:as-arrays? true描述clojure.java.jdbc/query的here功能。

我认为这有点帮助，但我可能仍然能够覆盖:result-set-fn到vec。

通过将所有行逻辑塞入:row-fn解决了核心问题。最初的OutOfMemory问题与迭代j/query结果集而不是定义特定的:row-fn有关。

新(工作)代码如下：

(defn -main []

(let [; {{{

db-spec local-postgres

source-sql "select * from public.f_5500 "

log-report-interval 1000

fetch-size 1000

row-count (atom 0)

field-delim "\u0001" ; unlikely to be in source feed,

; although i should still check in

; replace-newline below (for when "\t"

; is used especially)

row-delim "\n" ; unless fixed-width, target doesn't

; support non-printable chars for recDelim like

db-connection (doto ( j/get-connection db-spec) (.setAutoCommit false))

statement (j/prepare-statement db-connection source-sql :fetch-size fetch-size :concurrency :read-only)

start (System/currentTimeMillis)

rate-calc (fn [r] (float (/ r (/ ( - (System/currentTimeMillis) start) 100))))

replace-newline (fn [s] (if (string? s) (clojure.string/replace s #"\n" " ") s))

row-fn (fn [v]

(swap! row-count inc)

(when (zero? (mod @row-count log-report-interval))

(info (format "wrote %d rows" @row-count))

(info (format "\trows/s %.2f" (rate-calc @row-count)))

(info (format "\tPercent Mem used %s " (memory-percent-used))))

(str (join field-delim (doall (map #(replace-newline %) v))) row-delim ))

]; }}}

(info "Started database table dump session...")

(with-open [^java.io.Writer wrtr (io/writer "./sql/output.txt")]

(j/query db-connection [statement] :as-arrays? true :row-fn

#(.write wrtr (row-fn %))))

(info (format "\t\t\tCompleted with %d rows" @row-count))

(info (format "\t\t\tCompleted in %s seconds" (float (/ (- (System/currentTimeMillis) start) 1000))))

(info (format "\t\t\tAverage rows/s %.2f" (rate-calc @row-count)))

nil)

)

我试验的其他事情(成功有限)涉及音色记录和关闭标准;我想知道如果使用REPL它可能会在显示回我的编辑器(vim壁炉)之前缓存结果，我不确定这是否利用了大量的内存。

另外，我使用(.freeMemory (java.lang.Runtime/getRuntime))在记忆中添加了记录部分。我对VisualVM并不熟悉并准确指出我的问题所在。

我很高兴现在的工作方式，感谢大家的帮助。