Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to save tds.Vec()? #31

Open
hfxunlp opened this issue May 12, 2017 · 0 comments
Open

How to save tds.Vec()? #31

hfxunlp opened this issue May 12, 2017 · 0 comments

Comments

@hfxunlp
Copy link

hfxunlp commented May 12, 2017

I have 700M+ json data, and want to save it with tds.Vec(), the data could be loaded into a tds.Vec() correctly, but while call torch.save(), the luajit breaks. Here is my code:

local tds = require("tds")

function conjson(fname)
	print("load:"..fname)
	local function convt(tin)
		local rsv=tds.Vec()
		for _,v in ipairs(tin) do
			rsv[_]=torch.IntTensor(v):reshape(#v, 1)
		end
		return rsv
	end
	local file=io.open(fname)
	local rs=tds.Vec()
	local lind=file:read("*l")
	local curd=1
	while lind do
		local data=json.decode(lind)
		local id, qd, td=unpack(data)
		rs[curd]=tds.Vec(convt(id), torch.IntTensor(qd):reshape(#qd, 1), convt(td))
		lind=file:read("*l")
		curd=curd+1
	end
	file:close()
	print("loaded")
	return rs
end

torch.save("train.t7", conjson("train.data"))

and it gives this output:

load:train.data
loaded  
/usr/local/bin/luajit: not enough memory
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant