Korean Job Discussion Forums

[billybrobby]

I got some questions about programming. Specifically about optimization and how to write a program without chewing up tons of memory. I'm making a program in PERL that will be processing a vast body of text (a collection of 288 books) and organizing it into giant tables of data, say, the size of several webster's dictionaries. And it will performing algorithms using the processed data that will take maybe a matter of days. So optimization is important.

My question is, how should I store this data? In a mysql database or just a file? Which is faster? I don't hardly know sh*t about the back-end of programming languages and how they access memory, nor how that compares with the way that a database does it.

An alternative to answering these questions would be to direct me to a forum where somebody could. thanks.

[Wrench]

Let the database engine do the work. Since by the sounds of it you are doing database type program. I would do a database type structure mainly because you can then easily run queries on the said results.

[tfunk]

http://forums.devshed.com/

If you want I can post your question under my own username.

It really depends on the type of queries you're likely to run on the data. Are there complex relationships between the different 'tables' of data? Is a table just one big chunk of data or is it sorted into rows etc.?

IMHO it has to do with the complexity of the query, and the amount of data.

flat file = not complex
rdms = complex, scalability

[jajdude]

billyb, I haven't a clue, but I bet some on the technology forum do. You might move your post there.