LOGIN
Rob Freeman
Label resister (Project Leader)
 · Los AngelesU.S.
Edit
Delete
Share
Report
GENERAL

I had a bunch of team member requests and basically accepted them all because I couldn’t see a way to communicate easily directly.

I would like to limit team applications to people doing specific tasks however.

Would people who are interested perhaps post a little in the general discussion here presenting their experience in AI, or way they think they might contribute. We can discuss the general algorithm here as far as possible, and think about concrete contributions after that.

1 like 
Like
Share
Load 1 previous comments
Rob Freeman
Label resister (Project Leader)
 · Los AngelesU.S.
Edit
Delete
Share
Report

OK here’s the main breakdown of needs at the moment (By the way, if the site maintainers are reading this, it is the second time I’ve written this list today. I posted it some hours ago and now can’t find it. I seem to have a problem with the interface swallowing posts):

  1. I would like better interactive display of graph output. I believe it is possible to output a “dot” file from python in a vector graphics form which can be used to make re-sizing the graph etc. possible interactively within the browser. If someone knows how to do that please say so in a team join request, and we can try it.

  2. Currently input is done by building a graph of adjacency relations between successive items of input (text at the moment), and that is then cast into a matrix. But it is memory hungry. I’m running out of memory for larger data sets. There must be a better way to do it.

  3. Eventually we will want this to run in parallel. If anyone has any experience putting matrix operations onto GPU’s I’d like to try that.

  4. As I said earlier, I’m going to keep work on the core algorithm closed for now. But I’ve already written quite a lot about this online and the general principles I’m hypothesizing are open. If anyone else can figure out a way to do it from those general principles, that will be great too. Maybe it will be better than mine. Let’s compete and see who can figure out the best way to do it. To be brief those general principles are that I think the important structure for perception and prediction will be hierarchies of sub-networks in the matrix of adjacency relations between observations of the world (currently just looking at texts and seeking language phrase structure.) Different inputs (each new sentence) will highlight different such hierarchies of sub-networks. The key task I see is to identify hierarchical sub-networks expanding away from each new input.
    Most simply: to find hierarchical sub-networks in the adjacency matrix we build.

0 like 
Rob Freeman
Label resister (Project Leader)
 · Los AngelesU.S.
Edit
Delete
Share
Report

An expansion of the second task in that list.

Here’s something which has just caused me to run out of memory. I have been building my matricies for natural language texts by reading adjacency information into a graph, and then casting it as a matrix. But memory usage is getting out of control for larger data sets. If someone could show me a more efficient way to build these matricies, that would help.

Here is the code I use now:

for line in file.readlines():
tmp = line.strip()
sentence = tmp.split(" ")
for i, source in enumerate(sentence):
for dest in sentence[i+1:]:
G.add_edge(source, dest)
#print “Adding edge between source:”, source, “and dest:”, dest

C = to_numpy_matrix(G)

Where to_numpy_matrix is a function I found online which does the conversion.

There must be better ways of doing this.

I also want to keep a dictionary of words associated with the matrix rows and columns, for interpretation later on. Currently I make it from the graph before conversion.

1 like 
Robert Lancer
Chief Technology Officer at Collaborizm
 · New YorkU.S.
Edit
Delete
Share
Report

Rob Freeman I’m the technology lead at Collaborizm, you certainly should not loose any posts! Please post to my wall with any issues directly, we also have a staging site which you can test bugs out a. I will message you directly with a link to that site.

Thanks!

0 like 
Sarthak Sharma
 · GuwahatiIndia
Edit
Delete
Share
Report

For that memory issue, perhaps you can try dumping the output matrix data to a file. And then combine all the files for further processing.

1 like 
Rob Freeman
Label resister (Project Leader)
 · Los AngelesU.S.
Edit
Delete
Share
Report

Sarthak Sharma Thanks. Yes. Maybe. I’m not sure if there’s any natural modularity to the graph which would allow me to break it up. In principle any word might form an edge with any word in the data at any time.

I might form a list of pairs, sort them, and form a graph in parts that way, it’s true.

I think the main flaw is that I go through a graph in the first place. It shouldn’t be necessary. And then converting it from a graph to a matrix I imagine python, or the conversion function, probably use multiples of the natural size of the data structure.

I’m just reading adjacency information into a matrix. It is just the best logic for an adjacency matrix in python with words as keys escapes me for now. Reading up, I can form them by stacking lists, do something fancy with “dictionaries”, or perhaps count words in my data and declare a numpy matrix with the correct dimensions in the first place.

I don’t think the problem is hard. It is just because I am learning python as I go, it takes time.

2 likes 
Bi Sh
Design Invention Research
 · KathmanduNepal
Edit
Delete
Share
Report

i not very sure about the algorithm… but may b, if its useful. we can focus on just the Change between previous n present n future Matrix… less processing, less memory… also if u incorporate some adaptive/predictive algorithm it again become more efficient… i dont know im making any sense… just focusing on the philosophy of it…

0 like 
Leave a reply...
DISCOVER
CHAT
ALERTS
-2
DISCUSSIONS
-1
FEED